+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210909/B0c --output-run DEEP/20210909/B0c/science#step0/20240317T040120Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.6031 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0085 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0120 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/pipeline_orig.qgraph DEEP/20210909/B0c/science#step0/20240317T040120Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 4.0802 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 12.3982 seconds; current memory usage: 0.301 Gibyte, delta: 0.123 Gibyte, peak delta: 0.111 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.301 Gibyte (main), 0.299 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0501 seconds; current memory usage: 0.301 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.301 Gibyte (main), 0.299 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 2480 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210909_B0c_science#step0_20240317T040120Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.4699 seconds; current memory usage: 0.301 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.301 Gibyte (main), 0.299 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 2482 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.8205 seconds; current memory usage: 0.304 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.304 Gibyte (main), 0.299 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 6bcfd8a4-fbfc-4624-a681-52c4daebaff1 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-17 04:01:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-17 04:01:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 04:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:01:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-17 04:01:57 proc_lsst.multi:170 [INFO] [multi] job_id 3544 proc_lsst.multi INFO: [multi] job_id 3544 2024-03-17 04:01:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:01:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:01:57 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 3544 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 3544 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 3544 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash f9b5fad45d6d5692b7db5a97d21889ed parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e105300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ad75f0e0-51a2-47ab-ad1d-9a8ba608627b_isrForCrosstalkSources_1031726_54.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ad75f0e0-51a2-47ab-ad1d-9a8ba608627b_isrForCrosstalkSources_1031726_54.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash b3f20a457979e0d004ab807500bfe91f parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/fcc22e3c-300f-464d-a0b9-c62d83c4ad57_isrForCrosstalkSources_1031714_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/fcc22e3c-300f-464d-a0b9-c62d83c4ad57_isrForCrosstalkSources_1031714_39.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 6779b2056fefb2f3ee481b3e0a7586fc parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e105940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e86b24ae-78d6-480f-bb2c-b97a933c1f54_isrForCrosstalkSources_1031703_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e86b24ae-78d6-480f-bb2c-b97a933c1f54_isrForCrosstalkSources_1031703_13.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 6df087bd5cccae0cbd467db4548fee2f parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e105800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/5a511f79-4777-4c80-8873-bf1909945bfb_isrForCrosstalkSources_1031726_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/5a511f79-4777-4c80-8873-bf1909945bfb_isrForCrosstalkSources_1031726_59.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 357272ce3a60ed8b870eadd0ce6d2688 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e104a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/391214f3-0e3b-4e9d-8b44-4ddf28950b1f_isrForCrosstalkSources_1031732_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/391214f3-0e3b-4e9d-8b44-4ddf28950b1f_isrForCrosstalkSources_1031732_21.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 8fa04819b975c92afb377c1cfad97545 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e105a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/54691db5-5817-4ea4-9c1f-4cf029786ae5_isrForCrosstalkSources_1031721_61.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/54691db5-5817-4ea4-9c1f-4cf029786ae5_isrForCrosstalkSources_1031721_61.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash a97faf93f584d0c7478f14733b3fffa5 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e105620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/42d90be2-854e-4beb-a528-40acfcd6c0b3_isrForCrosstalkSources_1031708_35.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/42d90be2-854e-4beb-a528-40acfcd6c0b3_isrForCrosstalkSources_1031708_35.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 0ab070c334d3ee97dc4de9ef37bcc7ca parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e105b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e109aa99-6b9b-4a5f-84a9-0ddb55766cee_isrForCrosstalkSources_1031727_59.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e109aa99-6b9b-4a5f-84a9-0ddb55766cee_isrForCrosstalkSources_1031727_59.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 3435e940d05aba31cc3c28a80f1f1016 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e104c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/38fc7250-ec9e-4ce4-8fa2-39f2110e18f2_isrForCrosstalkSources_1031720_20.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/38fc7250-ec9e-4ce4-8fa2-39f2110e18f2_isrForCrosstalkSources_1031720_20.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash df98fab29598bfc65007b10fa3ed131c parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e105d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/ed3444fe-d064-427f-bfb6-8a0713744bf2_isrForCrosstalkSources_1031730_42.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/ed3444fe-d064-427f-bfb6-8a0713744bf2_isrForCrosstalkSources_1031730_42.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash a1b898efac2c4bc4d246f10d8f9dec67 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1054e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/328d0662-866f-4033-897b-b1a102bd9e87_isrForCrosstalkSources_1031712_4.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/328d0662-866f-4033-897b-b1a102bd9e87_isrForCrosstalkSources_1031712_4.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 48501ef9f3e3ee956fe11edc4598c8b8 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1059e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ea54b745-c64d-4d9b-9deb-605005b8f31a_isrForCrosstalkSources_1031729_42.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ea54b745-c64d-4d9b-9deb-605005b8f31a_isrForCrosstalkSources_1031729_42.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 61c3a88b8c412abd339f686c4ab13e33 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e105f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/73d22196-a6ed-4c86-b7ad-7c735c5a1be1_isrForCrosstalkSources_1031728_49.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/73d22196-a6ed-4c86-b7ad-7c735c5a1be1_isrForCrosstalkSources_1031728_49.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash c39ad8a36a51b659d2e62f29fad69fb8 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fe4aa082-816f-45cc-8fd0-cac0fac0f196_isrForCrosstalkSources_1031723_57.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fe4aa082-816f-45cc-8fd0-cac0fac0f196_isrForCrosstalkSources_1031723_57.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash c6d25e0a4aa848653b37b5fad40cbff1 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4b170816-0ab1-4e89-9ab6-326a27076be5_isrForCrosstalkSources_1031723_40.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4b170816-0ab1-4e89-9ab6-326a27076be5_isrForCrosstalkSources_1031723_40.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 958f15b48deb95943a09039fbacb888b parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e105ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/186f677c-c6ab-4710-bd33-2603b21047ba_isrForCrosstalkSources_1031722_56.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/186f677c-c6ab-4710-bd33-2603b21047ba_isrForCrosstalkSources_1031722_56.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 335f76650980c7089a5dd03aaa17820f parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/69b6adab-0cf7-4c8b-a9a8-bfb1941f9bd0_isrForCrosstalkSources_1031733_40.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/69b6adab-0cf7-4c8b-a9a8-bfb1941f9bd0_isrForCrosstalkSources_1031733_40.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 1bd49612de92748a2b1554b146f63613 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1065c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bbd5630d-f6d4-4a54-957b-6c3972d48d76_isrForCrosstalkSources_1031734_52.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bbd5630d-f6d4-4a54-957b-6c3972d48d76_isrForCrosstalkSources_1031734_52.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash d7fd24cbf854e3fcdb076b43e92fcfac parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/65e0eb97-13d4-438d-910b-f9b4699b03c4_isrForCrosstalkSources_1031711_29.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/65e0eb97-13d4-438d-910b-f9b4699b03c4_isrForCrosstalkSources_1031711_29.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 9590097fc0cf2a53d37a8518d6692f17 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/359e5de3-6aac-4d34-99f3-f507f6af16a0_isrForCrosstalkSources_1031706_46.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/359e5de3-6aac-4d34-99f3-f507f6af16a0_isrForCrosstalkSources_1031706_46.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 1510d1040c7ee01221ffbd12e6063f79 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/08338d78-fe07-4f7c-acef-40fbc6490471_isrForCrosstalkSources_1031732_24.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/08338d78-fe07-4f7c-acef-40fbc6490471_isrForCrosstalkSources_1031732_24.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 65080ce4acff022c7e2110ee47eb7ffc parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d85bfa30-01c0-4434-b9a3-7dc1df711e81_isrForCrosstalkSources_1031704_9.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d85bfa30-01c0-4434-b9a3-7dc1df711e81_isrForCrosstalkSources_1031704_9.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 319f830bffac7ab3d5b7b95b43ee01b2 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/389fe42a-eab8-4114-a589-de23385f57ab_isrForCrosstalkSources_1031738_25.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/389fe42a-eab8-4114-a589-de23385f57ab_isrForCrosstalkSources_1031738_25.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 91131e2f7943013735898858aae03703 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/60161c13-a619-40f9-8e61-5a38f589f87c_isrForCrosstalkSources_1031730_57.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/60161c13-a619-40f9-8e61-5a38f589f87c_isrForCrosstalkSources_1031730_57.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 07d6ecf9800ea2a1168b37896aa1360d parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1067a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fac45196-306a-4d63-b9a6-8784b1cb1509_isrForCrosstalkSources_1031707_37.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fac45196-306a-4d63-b9a6-8784b1cb1509_isrForCrosstalkSources_1031707_37.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 82b679cd46e90bc7be81ad0363c16aa4 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/97e0e125-8de8-4ffa-93f1-78b1d69e606b_isrForCrosstalkSources_1031718_39.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/97e0e125-8de8-4ffa-93f1-78b1d69e606b_isrForCrosstalkSources_1031718_39.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash b393cf0cb92439fd5c17c98cff0f751a parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/c77cb30f-bff6-4d7b-8079-7dad33b2d45d_isrForCrosstalkSources_1031735_24.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/c77cb30f-bff6-4d7b-8079-7dad33b2d45d_isrForCrosstalkSources_1031735_24.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 00899f80e40d81be6b71b69ae197b671 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/2c34f7d0-68b9-4ed4-add6-078db2c929bf_isrForCrosstalkSources_1031733_26.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/2c34f7d0-68b9-4ed4-add6-078db2c929bf_isrForCrosstalkSources_1031733_26.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 6d6ad5f53a733117ece208d2504f128c parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/5123f9c5-cdf8-4bdf-874b-91ed685ba162_isrForCrosstalkSources_1031702_44.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/5123f9c5-cdf8-4bdf-874b-91ed685ba162_isrForCrosstalkSources_1031702_44.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash dc0c15a6783683b5e08d83c0fb0e7d5a parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d275a19c-0176-4ddf-9692-15fdbbb37145_isrForCrosstalkSources_1031722_3.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d275a19c-0176-4ddf-9692-15fdbbb37145_isrForCrosstalkSources_1031722_3.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 84ef2834825c5f171d10019fec0709d0 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/101f16e4-64f1-4945-9517-f4df847c3648_isrForCrosstalkSources_1031707_8.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/101f16e4-64f1-4945-9517-f4df847c3648_isrForCrosstalkSources_1031707_8.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash c46566f38d5f9f86545e2132efa28e9e parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0144e49d-2658-4678-9af1-c15a777d2cb5_isrForCrosstalkSources_1031706_36.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0144e49d-2658-4678-9af1-c15a777d2cb5_isrForCrosstalkSources_1031706_36.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 854043bd57cbc937b167fa429ef11bd8 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/7a2c9bc3-08d6-41e0-b23b-14483e0ad157_isrForCrosstalkSources_1031715_1.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/7a2c9bc3-08d6-41e0-b23b-14483e0ad157_isrForCrosstalkSources_1031715_1.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 18ef61918fad723274629bfb132314b6 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/df5ebb1d-5738-4db4-a38b-ed869f6b52a3_isrForCrosstalkSources_1031724_44.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/df5ebb1d-5738-4db4-a38b-ed869f6b52a3_isrForCrosstalkSources_1031724_44.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 9bdf1f680c7306cd616e5808a18766cc parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/c5466470-615e-4c44-8973-4fc8dd781e2f_isrForCrosstalkSources_1031724_5.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/c5466470-615e-4c44-8973-4fc8dd781e2f_isrForCrosstalkSources_1031724_5.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 1fb45831ed9010834caa434cd3c33d7c parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/41c7cb66-83e4-492a-8fe5-b70a9a9d9721_isrForCrosstalkSources_1031708_23.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/41c7cb66-83e4-492a-8fe5-b70a9a9d9721_isrForCrosstalkSources_1031708_23.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 7f2ced48cd8f8925e6f1da9e8569ece6 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0433a00a-e723-400e-9803-ab2068206017_isrForCrosstalkSources_1031714_22.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0433a00a-e723-400e-9803-ab2068206017_isrForCrosstalkSources_1031714_22.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 16053197672ea7601fcf5a493655bdc6 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8be06254-36b6-4377-b6d3-5ee1095a64b1_isrForCrosstalkSources_1031732_34.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8be06254-36b6-4377-b6d3-5ee1095a64b1_isrForCrosstalkSources_1031732_34.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 2c33875519b51dfe76f6cdcb927c83cd parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6d67fb17-77e0-4536-957a-d4d7df5d96c2_isrForCrosstalkSources_1031737_15.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6d67fb17-77e0-4536-957a-d4d7df5d96c2_isrForCrosstalkSources_1031737_15.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 65ba70137ba20ccb1d8d5f895d137d04 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/34ca5f48-d1fe-4ec7-93e8-42740c4e0a4b_isrForCrosstalkSources_1031717_21.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/34ca5f48-d1fe-4ec7-93e8-42740c4e0a4b_isrForCrosstalkSources_1031717_21.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 5259ae2040126e0e9acaf4dd17115d6f parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b3e9beff-b14c-4e45-8b31-fd6bb4e52e3c_isrForCrosstalkSources_1031721_14.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b3e9beff-b14c-4e45-8b31-fd6bb4e52e3c_isrForCrosstalkSources_1031721_14.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash e160de82112de6126ca54944e6737768 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/08a4c632-a4bd-43e9-9b07-c36d382cc145_isrForCrosstalkSources_1031728_28.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/08a4c632-a4bd-43e9-9b07-c36d382cc145_isrForCrosstalkSources_1031728_28.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 0a917998773435c5a7f09f8a0a140fc2 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/e76be664-4ce3-41e7-a703-1ed4925ad120_isrForCrosstalkSources_1031726_37.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/e76be664-4ce3-41e7-a703-1ed4925ad120_isrForCrosstalkSources_1031726_37.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 1177f22a490d6d73aa3230b7b5e50067 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d85985e2-8754-4c2e-b3e3-c5354b9fdef3_isrForCrosstalkSources_1031714_21.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d85985e2-8754-4c2e-b3e3-c5354b9fdef3_isrForCrosstalkSources_1031714_21.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash b13212c31d0b9638af4576b86d41c610 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4fd6d74d-34ea-42be-bb9d-b48781448a96_isrForCrosstalkSources_1031703_7.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4fd6d74d-34ea-42be-bb9d-b48781448a96_isrForCrosstalkSources_1031703_7.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 57dd832035ea240010a5d2667c4c65ec parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2b1db993-5303-4771-8721-e652d95b8514_isrForCrosstalkSources_1031703_21.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2b1db993-5303-4771-8721-e652d95b8514_isrForCrosstalkSources_1031703_21.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash cdf60f7ae8f9317bebca5a641baac1d8 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4ed70c82-25bf-4d29-be9b-8f089925774d_isrForCrosstalkSources_1031723_56.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4ed70c82-25bf-4d29-be9b-8f089925774d_isrForCrosstalkSources_1031723_56.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 1d49cc28f8ae01c29fb0f7878d20064d parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a0aaa727-ac70-4b60-9708-efe18aa9c584_isrForCrosstalkSources_1031725_11.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a0aaa727-ac70-4b60-9708-efe18aa9c584_isrForCrosstalkSources_1031725_11.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 4ae7c71ce9bc88ee31c5b094f0c6e4de parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/dde40872-abf3-4aff-b47b-758776c4087f_isrForCrosstalkSources_1031706_33.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/dde40872-abf3-4aff-b47b-758776c4087f_isrForCrosstalkSources_1031706_33.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 37e44537bf3e7bad5331023e47145e2b parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/57190881-7df4-40fa-9e5e-ab5881d058cb_isrForCrosstalkSources_1031704_24.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/57190881-7df4-40fa-9e5e-ab5881d058cb_isrForCrosstalkSources_1031704_24.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 483b286bfb1f0f6da98f4e22bf2dd330 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/6ebb85a0-8c0a-4fb0-a91b-ead7390be316_isrForCrosstalkSources_1031705_53.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/6ebb85a0-8c0a-4fb0-a91b-ead7390be316_isrForCrosstalkSources_1031705_53.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 683388fb4901c39640477ae7b78c176c parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/52c4a6b0-5455-46b0-8105-c2377b4aab1f_isrForCrosstalkSources_1031716_37.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/52c4a6b0-5455-46b0-8105-c2377b4aab1f_isrForCrosstalkSources_1031716_37.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 3dcf9537aba6f635c7bd973cd68f7f63 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/26a13790-ec24-4211-904c-06cfb0ed9024_isrForCrosstalkSources_1031712_31.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/26a13790-ec24-4211-904c-06cfb0ed9024_isrForCrosstalkSources_1031712_31.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash aef1a1d791a31f0dd8e7d1142e015d14 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/cde51801-d464-40f1-84d3-38eaa02d5900_isrForCrosstalkSources_1031704_16.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/cde51801-d464-40f1-84d3-38eaa02d5900_isrForCrosstalkSources_1031704_16.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 243c36322a30966b08f05dfb07c8e96d parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/793f3a52-291f-4b2e-9760-391472ce2734_isrForCrosstalkSources_1031713_9.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/793f3a52-291f-4b2e-9760-391472ce2734_isrForCrosstalkSources_1031713_9.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 116863de96532ab1b3ace7f9832e6545 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1074c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e772b3fc-e666-4d85-ab79-7aaee698d466_isrForCrosstalkSources_1031707_19.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e772b3fc-e666-4d85-ab79-7aaee698d466_isrForCrosstalkSources_1031707_19.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 51048fd31cc1c46351b082296d3ffc8c parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c17fe010-0f34-4b79-ad6f-2252b761e4d3_isrForCrosstalkSources_1031716_59.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c17fe010-0f34-4b79-ad6f-2252b761e4d3_isrForCrosstalkSources_1031716_59.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash d60329877d4a4f1209e23c74937fafd8 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4f3600a9-5794-4c43-b7af-c45207c8dea4_isrForCrosstalkSources_1031733_28.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4f3600a9-5794-4c43-b7af-c45207c8dea4_isrForCrosstalkSources_1031733_28.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 7033933d5c27307af5625cf5f6a99ca4 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/ca6b8fcc-0a40-4c06-80d7-f283656f4071_isrForCrosstalkSources_1031716_40.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/ca6b8fcc-0a40-4c06-80d7-f283656f4071_isrForCrosstalkSources_1031716_40.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 84cc972fb59e0ce6cf43b7c5108471f0 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e1072e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a0500995-2190-4aba-b168-378f1f5eeadb_isrForCrosstalkSources_1031720_58.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a0500995-2190-4aba-b168-378f1f5eeadb_isrForCrosstalkSources_1031720_58.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 1eea86f8229babe4784b059bb63e9694 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e106520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/176fc966-eee8-4f62-ad08-77f3d093c5bd_isrForCrosstalkSources_1031726_2.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/176fc966-eee8-4f62-ad08-77f3d093c5bd_isrForCrosstalkSources_1031726_2.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 6f189d86246c7f425f30ac8c5b4ee82f parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/84575da9-6d89-46db-88f4-b6a26b3b09de_isrForCrosstalkSources_1031736_26.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/84575da9-6d89-46db-88f4-b6a26b3b09de_isrForCrosstalkSources_1031736_26.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 6403d76e8dfa56eefe6178d5a983af82 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6bd867a4-eb1a-433a-9b9c-ce961ad3e6f5_isrForCrosstalkSources_1031728_31.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6bd867a4-eb1a-433a-9b9c-ce961ad3e6f5_isrForCrosstalkSources_1031728_31.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 06b3ba05a06663952d528bc04a3958c8 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1b827745-2e94-4777-8e51-e963e770e523_isrForCrosstalkSources_1031700_22.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1b827745-2e94-4777-8e51-e963e770e523_isrForCrosstalkSources_1031700_22.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash da96c3040e011be509255fd10300fea7 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fdc12190-7817-46fd-b8af-865e969a28c1_isrForCrosstalkSources_1031700_42.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fdc12190-7817-46fd-b8af-865e969a28c1_isrForCrosstalkSources_1031700_42.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 8ba4365831edb54978af767cad5ef1cb parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cacb828e-c2ec-4518-847e-cb1fa69a97fa_isrForCrosstalkSources_1031728_61.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cacb828e-c2ec-4518-847e-cb1fa69a97fa_isrForCrosstalkSources_1031728_61.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash e5a62f22cd38bbfe1b138b843f337af6 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/2795895c-8c9f-49b2-9a58-3d6ba9135353_isrForCrosstalkSources_1031738_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/2795895c-8c9f-49b2-9a58-3d6ba9135353_isrForCrosstalkSources_1031738_2.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 7d0e970bace5bde2bf936302f298c802 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e1881e81-6036-4a7c-8667-4f5a50092cdc_isrForCrosstalkSources_1031725_52.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e1881e81-6036-4a7c-8667-4f5a50092cdc_isrForCrosstalkSources_1031725_52.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash ef1d7bc5b5c7cf60b5da7ca6f2326b1c parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/13d0dd3f-6f20-4de8-91a4-396785d711b0_isrForCrosstalkSources_1031733_12.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/13d0dd3f-6f20-4de8-91a4-396785d711b0_isrForCrosstalkSources_1031733_12.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 87834c9da1b84408dd656a54a4fa46b6 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f17bd0c7-9767-4a72-a11e-f0592e8ead29_isrForCrosstalkSources_1031735_40.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f17bd0c7-9767-4a72-a11e-f0592e8ead29_isrForCrosstalkSources_1031735_40.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 7a23a1657db3b5ac0dc2b92974762a9b parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/68aee9ea-c24d-4040-ab31-cfd2f8ac3edc_isrForCrosstalkSources_1031721_34.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/68aee9ea-c24d-4040-ab31-cfd2f8ac3edc_isrForCrosstalkSources_1031721_34.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 9f589137e52d19aba937913a8fca51af parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/2a7384d5-4ca0-443b-b331-ed67eb4c5f11_isrForCrosstalkSources_1031714_3.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/2a7384d5-4ca0-443b-b331-ed67eb4c5f11_isrForCrosstalkSources_1031714_3.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash b430e3c26d937515085c9eb8986c09ff parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/304edd5d-bebd-4aa5-b310-ae0c88c40bc9_isrForCrosstalkSources_1031704_49.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/304edd5d-bebd-4aa5-b310-ae0c88c40bc9_isrForCrosstalkSources_1031704_49.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 89e7362bad54105a72e83bed9b25177a parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/765ce3ef-9192-44b2-aff4-54f1ef2a7b50_isrForCrosstalkSources_1031726_52.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/765ce3ef-9192-44b2-aff4-54f1ef2a7b50_isrForCrosstalkSources_1031726_52.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash c7f51d97f08ac0a3e69ba2c7b0bfdc02 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/efd0f4ac-9a1e-427c-978f-80592b102ce5_isrForCrosstalkSources_1031705_3.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/efd0f4ac-9a1e-427c-978f-80592b102ce5_isrForCrosstalkSources_1031705_3.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 3bf2a05fcdb2a3a8fc8de0cb4d7e6627 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ee21d936-a0a6-4c09-92b8-bd07b7e98082_isrForCrosstalkSources_1031737_2.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ee21d936-a0a6-4c09-92b8-bd07b7e98082_isrForCrosstalkSources_1031737_2.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash b937ebee0a3a4c9555bfea0a300a6564 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/66b1e396-c38c-4737-a532-1cf985952736_isrForCrosstalkSources_1031712_7.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/66b1e396-c38c-4737-a532-1cf985952736_isrForCrosstalkSources_1031712_7.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 64d0234798ef8e714d1b00f3bd0c9d1e parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/51135140-f408-49cd-a8f7-2a4fd2863291_isrForCrosstalkSources_1031702_52.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/51135140-f408-49cd-a8f7-2a4fd2863291_isrForCrosstalkSources_1031702_52.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 5fd4976132de0f041e4b7d718bccbe20 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4145b74f-0991-4fa6-bf19-9ddc53e163d3_isrForCrosstalkSources_1031715_57.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4145b74f-0991-4fa6-bf19-9ddc53e163d3_isrForCrosstalkSources_1031715_57.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash df4cb016ed6fdbd0c88d18663ce77392 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/26f2aa33-df17-4d62-be64-b951768635ff_isrForCrosstalkSources_1031710_59.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/26f2aa33-df17-4d62-be64-b951768635ff_isrForCrosstalkSources_1031710_59.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash fca282816f3cfc05e34a41708a9e1ff7 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/73b24f5c-1014-4655-87be-aa2a91144e5b_isrForCrosstalkSources_1031719_54.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/73b24f5c-1014-4655-87be-aa2a91144e5b_isrForCrosstalkSources_1031719_54.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash b0e47625a6ad0a99d3b901a740cb7f24 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/dac248eb-bbe4-481f-9745-b51c46c31ff8_isrForCrosstalkSources_1031708_3.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/dac248eb-bbe4-481f-9745-b51c46c31ff8_isrForCrosstalkSources_1031708_3.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash ee4948d189d226ba9c38bdd3ba2fee3f parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/80ecf0e6-fc37-41e4-a9d5-91ed301731e8_isrForCrosstalkSources_1031720_57.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/80ecf0e6-fc37-41e4-a9d5-91ed301731e8_isrForCrosstalkSources_1031720_57.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash c62324202a0574b5f975ac29c14aa1b3 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/f43382ff-d5b6-456a-b303-b07c18df359e_isrForCrosstalkSources_1031718_20.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/f43382ff-d5b6-456a-b303-b07c18df359e_isrForCrosstalkSources_1031718_20.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash c6a87726dd647406d0162fbbbe423de2 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/4c517d0f-5a9d-4bf6-b145-032fecf4e16a_isrForCrosstalkSources_1031722_54.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/4c517d0f-5a9d-4bf6-b145-032fecf4e16a_isrForCrosstalkSources_1031722_54.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash db6967f4f7cfdeec1d3d1725ec10af01 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fdd2240b-f1e0-41b7-af8a-bfc3e4e4f33b_isrForCrosstalkSources_1031732_47.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fdd2240b-f1e0-41b7-af8a-bfc3e4e4f33b_isrForCrosstalkSources_1031732_47.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 1844e576f7dbc1607d82b127897cbfc9 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/9e65467a-225d-45dc-8003-27116bfb122d_isrForCrosstalkSources_1031727_19.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/9e65467a-225d-45dc-8003-27116bfb122d_isrForCrosstalkSources_1031727_19.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 9ac06d0fa1471e898eeee21575f1d1ce parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/349fbca3-9f48-4425-a3ec-2eaa954f8006_isrForCrosstalkSources_1031722_10.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/349fbca3-9f48-4425-a3ec-2eaa954f8006_isrForCrosstalkSources_1031722_10.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 5d9fa4e4a72a313fdd14ebb906e70269 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/e01ea1f7-84bd-4082-b7fa-2a442a699cf8_isrForCrosstalkSources_1031699_15.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/e01ea1f7-84bd-4082-b7fa-2a442a699cf8_isrForCrosstalkSources_1031699_15.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 7a2c2005f8f00926e9543b339be04faf parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/171647c4-75ce-4f6c-9735-fbc0f4d765df_isrForCrosstalkSources_1031730_3.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/171647c4-75ce-4f6c-9735-fbc0f4d765df_isrForCrosstalkSources_1031730_3.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash af5eec94e2f538a96614a239838d7aae parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/6b8f6538-0662-435c-8f22-239c3c95c64d_isrForCrosstalkSources_1031725_14.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/6b8f6538-0662-435c-8f22-239c3c95c64d_isrForCrosstalkSources_1031725_14.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash cf1e3f989304e7c17a2a60baf32d2bc4 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a014b8b0-3760-4e8a-aa32-d938b1ea0cd0_isrForCrosstalkSources_1031728_30.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a014b8b0-3760-4e8a-aa32-d938b1ea0cd0_isrForCrosstalkSources_1031728_30.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 73d8c5304fe6a8fb992d4326384e8b69 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/ff14a77b-a283-42e5-a04e-bda20e7269e5_isrForCrosstalkSources_1031735_45.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/ff14a77b-a283-42e5-a04e-bda20e7269e5_isrForCrosstalkSources_1031735_45.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 8e8187aeb0c1c5d7e84bcf580677c905 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/26b2b291-9c80-484f-98b7-8e622270fecd_isrForCrosstalkSources_1031722_37.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/26b2b291-9c80-484f-98b7-8e622270fecd_isrForCrosstalkSources_1031722_37.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash b23a81cbe3c5f92c5d96208fb80de591 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/82b2ba86-91c2-4f53-b030-eb1005d62696_isrForCrosstalkSources_1031712_46.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/82b2ba86-91c2-4f53-b030-eb1005d62696_isrForCrosstalkSources_1031712_46.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 9bb49e49dc3a402ca130afaeb489a57b parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5fcf73bd-6f7e-4fae-86c3-dd9f0df0c3c9_isrForCrosstalkSources_1031724_47.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5fcf73bd-6f7e-4fae-86c3-dd9f0df0c3c9_isrForCrosstalkSources_1031724_47.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 1827bdd27496b8e432a6f55d62153edb parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6ca16a7b-685e-4c23-a4ad-4abe942ec01d_isrForCrosstalkSources_1031711_35.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6ca16a7b-685e-4c23-a4ad-4abe942ec01d_isrForCrosstalkSources_1031711_35.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 5b956e03d1e87f31b9cd533f997aac7a parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e1238eab-0a04-4858-9eee-897dbbb5be5a_isrForCrosstalkSources_1031710_25.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e1238eab-0a04-4858-9eee-897dbbb5be5a_isrForCrosstalkSources_1031710_25.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 4a4a08c4ee312243325b6e578ca00631 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/850714ec-3520-40a6-8c01-a0119fed5c02_isrForCrosstalkSources_1031733_32.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/850714ec-3520-40a6-8c01-a0119fed5c02_isrForCrosstalkSources_1031733_32.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 6ec4dba906b77d1f23db86dfdbc01289 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/3ac2128f-4874-44ea-88ae-f837263099f7_isrForCrosstalkSources_1031704_51.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/3ac2128f-4874-44ea-88ae-f837263099f7_isrForCrosstalkSources_1031704_51.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash b2c0f61f9e6fd62e9351700fe023aa57 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/03808966-6af9-4840-988e-1367e25abf52_isrForCrosstalkSources_1031730_14.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/03808966-6af9-4840-988e-1367e25abf52_isrForCrosstalkSources_1031730_14.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 3371386bbf2ee46a145bc6d19b578bcf parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ad1784b4-c7d0-40b3-8431-2f70d865bb8f_isrForCrosstalkSources_1031717_42.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ad1784b4-c7d0-40b3-8431-2f70d865bb8f_isrForCrosstalkSources_1031717_42.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 1f22338f0b2ff6f17d6c55d6aca29dba parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/53370d68-ba9e-47cc-a669-ff40561222e4_isrForCrosstalkSources_1031732_32.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/53370d68-ba9e-47cc-a669-ff40561222e4_isrForCrosstalkSources_1031732_32.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash ead527d4a749412f9216159d44f5fb38 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/93689b0f-907f-42b6-9418-e7ef72530b86_isrForCrosstalkSources_1031730_22.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/93689b0f-907f-42b6-9418-e7ef72530b86_isrForCrosstalkSources_1031730_22.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash eb194b150e62a7c70545d3f930f1356b parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0fc8cce6-251e-482d-9e61-968f4c3a1fef_isrForCrosstalkSources_1031717_6.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0fc8cce6-251e-482d-9e61-968f4c3a1fef_isrForCrosstalkSources_1031717_6.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 09864999fafb658a7a7642ff95e5d973 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2e54a598-3267-4d82-a00d-1dd0da717540_isrForCrosstalkSources_1031704_53.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2e54a598-3267-4d82-a00d-1dd0da717540_isrForCrosstalkSources_1031704_53.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 8d484636e0a657045b2de4b513dd5cc0 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/e7606d77-3625-4973-9b89-f62ebf87d594_isrForCrosstalkSources_1031721_46.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/e7606d77-3625-4973-9b89-f62ebf87d594_isrForCrosstalkSources_1031721_46.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash b567c3df9b749427a181aad6e140fdde parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/63e5c995-8288-495f-927c-5a472840d661_isrForCrosstalkSources_1031701_20.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/63e5c995-8288-495f-927c-5a472840d661_isrForCrosstalkSources_1031701_20.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 1869453cc6e99f6ee41f0222a66c1bf2 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4b3a4136-8834-4b70-8c25-6e28f4e8c5e0_isrForCrosstalkSources_1031716_21.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4b3a4136-8834-4b70-8c25-6e28f4e8c5e0_isrForCrosstalkSources_1031716_21.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash d4fa17d2c65a16ef07980654a52e47c1 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1cd64e75-ee74-45e1-a954-57e1f08cb262_isrForCrosstalkSources_1031713_11.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1cd64e75-ee74-45e1-a954-57e1f08cb262_isrForCrosstalkSources_1031713_11.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash c7998d1f56610e45d3105a5351526b92 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4c8cdd8f-95c3-4f7b-acba-12adf2a9058a_isrForCrosstalkSources_1031737_42.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4c8cdd8f-95c3-4f7b-acba-12adf2a9058a_isrForCrosstalkSources_1031737_42.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 8b352c3cac5a051e6692eb018df5f141 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/81f71d40-aaf8-4568-9f8c-31138440d617_isrForCrosstalkSources_1031727_8.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/81f71d40-aaf8-4568-9f8c-31138440d617_isrForCrosstalkSources_1031727_8.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 822f79575defae3b792f43d989ad702a parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/00103313-056a-46d5-8b04-3fac4fc70ad9_isrForCrosstalkSources_1031706_40.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/00103313-056a-46d5-8b04-3fac4fc70ad9_isrForCrosstalkSources_1031706_40.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash d8fa685b280558eef8d083b52dff38be parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/af153a57-ec3f-44a7-b1f0-167504addc1d_isrForCrosstalkSources_1031715_41.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/af153a57-ec3f-44a7-b1f0-167504addc1d_isrForCrosstalkSources_1031715_41.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash a7594ecfbed93790c1a91c44f0493968 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/0d23a2e7-80ba-4a08-85f1-ffb5c528f140_isrForCrosstalkSources_1031700_18.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/0d23a2e7-80ba-4a08-85f1-ffb5c528f140_isrForCrosstalkSources_1031700_18.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 443004bb6aac2cf3777aaee7244b0bb2 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c89fc29c-3da9-46dd-b72a-ab9fef044a6a_isrForCrosstalkSources_1031710_13.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c89fc29c-3da9-46dd-b72a-ab9fef044a6a_isrForCrosstalkSources_1031710_13.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 28656823fac51e2d7ce3949f68a4eaaf parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/92f4fb91-ffbf-4047-96c5-2926aa607f2a_isrForCrosstalkSources_1031718_37.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/92f4fb91-ffbf-4047-96c5-2926aa607f2a_isrForCrosstalkSources_1031718_37.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 0cd730b6d26cc562343b2604c208388b parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fdfca7c1-f1f4-4dbf-a226-38ad7c165351_isrForCrosstalkSources_1031723_33.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fdfca7c1-f1f4-4dbf-a226-38ad7c165351_isrForCrosstalkSources_1031723_33.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 105ad82e9363f71617b63f6f2d6d67f7 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/4e2f7527-d01a-4f39-b624-3c5371bd67af_isrForCrosstalkSources_1031710_5.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/4e2f7527-d01a-4f39-b624-3c5371bd67af_isrForCrosstalkSources_1031710_5.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 4cdf63d5921cf02f4347dd90f4bc79ef parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/55f9357d-bc22-4d82-b810-0d755189d567_isrForCrosstalkSources_1031737_31.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/55f9357d-bc22-4d82-b810-0d755189d567_isrForCrosstalkSources_1031737_31.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 5f1509d8f03f1dda4e8d5bccb758cf57 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2870c680-dda5-4dd5-912b-fb1df87972dc_isrForCrosstalkSources_1031727_60.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2870c680-dda5-4dd5-912b-fb1df87972dc_isrForCrosstalkSources_1031727_60.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 65fa9b168462567a96fe2218c841506c parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/89e71966-c38c-4635-ab56-5c59d29dcfc6_isrForCrosstalkSources_1031722_41.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/89e71966-c38c-4635-ab56-5c59d29dcfc6_isrForCrosstalkSources_1031722_41.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 3db81ea76b43d709d63aec9816c8050a parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/61147069-14fe-4957-9fc2-de548f82b6cf_isrForCrosstalkSources_1031716_23.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/61147069-14fe-4957-9fc2-de548f82b6cf_isrForCrosstalkSources_1031716_23.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash a5516ce05eb8ae11e0fbadec260f6275 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/01df9b3d-ab9f-4b27-954b-40ce5c93c868_isrForCrosstalkSources_1031725_43.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/01df9b3d-ab9f-4b27-954b-40ce5c93c868_isrForCrosstalkSources_1031725_43.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 786c49dd889b680272a77d4e7b0738cb parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/8af6e640-f46e-4495-8f53-99c2b668dce4_isrForCrosstalkSources_1031737_33.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/8af6e640-f46e-4495-8f53-99c2b668dce4_isrForCrosstalkSources_1031737_33.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 949067ceb84409d89254d52420792b32 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/db1935b7-b725-40df-99c2-2153cb95da85_isrForCrosstalkSources_1031700_7.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/db1935b7-b725-40df-99c2-2153cb95da85_isrForCrosstalkSources_1031700_7.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 5fb6f8d029494c8b1a9d848e2aade360 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/86727ae2-a631-4a68-8534-3ad7e026ee54_isrForCrosstalkSources_1031732_20.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/86727ae2-a631-4a68-8534-3ad7e026ee54_isrForCrosstalkSources_1031732_20.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash b469416942e382a87cb5cc3b60476fe3 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4faac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/a15ccee3-a972-47b2-80cc-993dca69a3fe_isrForCrosstalkSources_1031710_37.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/a15ccee3-a972-47b2-80cc-993dca69a3fe_isrForCrosstalkSources_1031710_37.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 0e8ec9b55da21c821622df98b3516958 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4faa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/5f6aef4e-69ec-4b25-97ff-a111e319c878_isrForCrosstalkSources_1031707_6.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/5f6aef4e-69ec-4b25-97ff-a111e319c878_isrForCrosstalkSources_1031707_6.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash bf21ff80ca0b034e38723ff67233f3dd parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8177bfa3-003c-4673-ac27-fe26c9706e01_isrForCrosstalkSources_1031699_42.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8177bfa3-003c-4673-ac27-fe26c9706e01_isrForCrosstalkSources_1031699_42.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 63e462f9f8b6b253ce8025e15ba391e6 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2e2c2e1e-ea2f-4f9e-b9bb-2868491444bd_isrForCrosstalkSources_1031734_27.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2e2c2e1e-ea2f-4f9e-b9bb-2868491444bd_isrForCrosstalkSources_1031734_27.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash a3c3b684f5c6b6a15ddcb153a6dbcb68 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4faca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/99c33805-eef5-4ea5-b331-139f7f2acc6c_isrForCrosstalkSources_1031700_17.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/99c33805-eef5-4ea5-b331-139f7f2acc6c_isrForCrosstalkSources_1031700_17.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash e3328d69ebfad71ccc62a0cb36ee5fbf parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/d7b87ed9-b1ec-4a8a-b2a0-7cc0ac22ad35_isrForCrosstalkSources_1031710_1.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/d7b87ed9-b1ec-4a8a-b2a0-7cc0ac22ad35_isrForCrosstalkSources_1031710_1.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash d997f45a984c15610ab75083bd3a107e parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d908bf00-fe86-4e32-8082-2b9514896ffd_isrForCrosstalkSources_1031737_10.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d908bf00-fe86-4e32-8082-2b9514896ffd_isrForCrosstalkSources_1031737_10.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 96e696583587878426548218ea7683b9 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/538809e8-303e-4f81-a6af-41c1775b45a6_isrForCrosstalkSources_1031726_57.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/538809e8-303e-4f81-a6af-41c1775b45a6_isrForCrosstalkSources_1031726_57.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash e947da3c83dbb359c6ba3a4d3ab9325f parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4faf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/ec92e6c3-7cad-4f5a-a861-baaf0f1ec6b9_isrForCrosstalkSources_1031723_27.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/ec92e6c3-7cad-4f5a-a861-baaf0f1ec6b9_isrForCrosstalkSources_1031723_27.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 2e7121151c00cc1968cacfb73bc6011b parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/6c0da8a4-a066-4ba8-bf3a-5d2023fcc470_isrForCrosstalkSources_1031713_28.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/6c0da8a4-a066-4ba8-bf3a-5d2023fcc470_isrForCrosstalkSources_1031713_28.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash ed9a38c72af0f800550aa59e2aa5f8f2 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/24b779b6-ebc9-423e-a258-297317e5afd3_isrForCrosstalkSources_1031700_35.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/24b779b6-ebc9-423e-a258-297317e5afd3_isrForCrosstalkSources_1031700_35.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 21f95e25ada5e1793998a17f2a7a9ef5 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4a9dca73-0ca0-47da-971e-e809614442ec_isrForCrosstalkSources_1031732_56.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4a9dca73-0ca0-47da-971e-e809614442ec_isrForCrosstalkSources_1031732_56.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 9093b72ffc3e0527f8736f8d7c6f7df4 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/470c4db5-36a0-48ab-a504-e1fe6980bc6a_isrForCrosstalkSources_1031735_32.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/470c4db5-36a0-48ab-a504-e1fe6980bc6a_isrForCrosstalkSources_1031735_32.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 2e539bcb00b3dc86f4661083d14c128b parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/b8bf4923-5c06-48c9-9b61-d355b35ffb61_isrForCrosstalkSources_1031710_51.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/b8bf4923-5c06-48c9-9b61-d355b35ffb61_isrForCrosstalkSources_1031710_51.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash ed918a4426b81ffac6c95082b24d270a parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c0360758-26fa-454a-ab12-3cf0b53d1758_isrForCrosstalkSources_1031703_52.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c0360758-26fa-454a-ab12-3cf0b53d1758_isrForCrosstalkSources_1031703_52.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash c32d4d189d51bc6f128970664ca14064 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8c8e3ecf-ddf8-4433-82ad-73820ce8401e_isrForCrosstalkSources_1031721_1.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8c8e3ecf-ddf8-4433-82ad-73820ce8401e_isrForCrosstalkSources_1031721_1.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 70987c838a5d0e854647b3b5d2a5a0ff parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:01:58 proc_lsst.multi:146 [INFO] found job 3544 in provider local proc_lsst.multi INFO: found job 3544 in provider local parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/eebe3a0a-a2bd-4c5c-b34e-5c7df383bebd_isrForCrosstalkSources_1031701_13.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/eebe3a0a-a2bd-4c5c-b34e-5c7df383bebd_isrForCrosstalkSources_1031701_13.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash f2d11a82d97ee9295897ae666bb1364d parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 144 parsl.dataflow.strategy DEBUG: Executor multi has 144 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 35 more blocks parsl.executors.status_handling INFO: Scaling out by 35 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-17 04:01:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-17 04:01:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:01:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:01:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:01:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:01:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 04:01:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:01:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/83f0c4b6-f814-4b0f-969f-ab6ccdcee944_isrForCrosstalkSources_1031724_30.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/83f0c4b6-f814-4b0f-969f-ab6ccdcee944_isrForCrosstalkSources_1031724_30.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 2e9a0f795081aa5f3268319288240954 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/75dae491-2bb8-45bb-8cc7-3952d35b53f6_isrForCrosstalkSources_1031717_35.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/75dae491-2bb8-45bb-8cc7-3952d35b53f6_isrForCrosstalkSources_1031717_35.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 3fb05abf0733ab3f13adc0722360bc06 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4f8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7220336c-3444-4ada-8b91-50ba0af32b00_isrForCrosstalkSources_1031714_11.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7220336c-3444-4ada-8b91-50ba0af32b00_isrForCrosstalkSources_1031714_11.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 0faf8d91854ed607c19653ba9e1ec249 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d12a0f7f-90c0-49aa-a90e-22757ab9b469_isrForCrosstalkSources_1031703_24.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d12a0f7f-90c0-49aa-a90e-22757ab9b469_isrForCrosstalkSources_1031703_24.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 5ba41316ece65e2298e7e1d01f8a3385 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4feb177d-73b6-45ca-876d-4999ee0fd38f_isrForCrosstalkSources_1031733_25.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4feb177d-73b6-45ca-876d-4999ee0fd38f_isrForCrosstalkSources_1031733_25.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 247ccccf0d615391560f8e10ff8767e7 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d2cada02-4be5-4563-96e1-7a6d461dd6a9_isrForCrosstalkSources_1031704_31.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d2cada02-4be5-4563-96e1-7a6d461dd6a9_isrForCrosstalkSources_1031704_31.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 29d66fcdb86ac84e5d0da985b868b7eb parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/44a287b7-6f07-4451-a858-92a51ac71f5f_isrForCrosstalkSources_1031712_42.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/44a287b7-6f07-4451-a858-92a51ac71f5f_isrForCrosstalkSources_1031712_42.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 287be13cc92bb8869bf71e3d6ad84bd3 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/927cf93e-5d16-4dab-a6a7-6f74e2fbf47f_isrForCrosstalkSources_1031725_35.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/927cf93e-5d16-4dab-a6a7-6f74e2fbf47f_isrForCrosstalkSources_1031725_35.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 38d9ad53b1258552f0c41cc8fcbe48aa parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0df69710-4383-490c-83a1-abdcbd6cf99b_isrForCrosstalkSources_1031720_13.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0df69710-4383-490c-83a1-abdcbd6cf99b_isrForCrosstalkSources_1031720_13.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash bd3bbfec0e94c23118cc43d8be1ff963 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9c6072fe-121b-437f-ab9e-757008e9a5df_isrForCrosstalkSources_1031730_17.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9c6072fe-121b-437f-ab9e-757008e9a5df_isrForCrosstalkSources_1031730_17.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 2037a4eb36b5cd5ca029d343035f965d parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/8bd91eae-ebfe-4c16-be47-6610a6c5612e_isrForCrosstalkSources_1031726_62.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/8bd91eae-ebfe-4c16-be47-6610a6c5612e_isrForCrosstalkSources_1031726_62.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 38aaca494ba43f83986079c5162e17f5 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eab7f269-99d5-46b9-bf3b-6500ca062c03_isrForCrosstalkSources_1031719_23.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eab7f269-99d5-46b9-bf3b-6500ca062c03_isrForCrosstalkSources_1031719_23.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash f18c5814dd2b83a0ed19737ba5a34af4 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ded29da8-f260-4c29-bba7-26558c24c08a_isrForCrosstalkSources_1031717_58.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ded29da8-f260-4c29-bba7-26558c24c08a_isrForCrosstalkSources_1031717_58.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash cabe8ae05461ef281401bc7293d9c963 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/fce6dc21-8290-4444-ad41-6f6d113affab_isrForCrosstalkSources_1031703_56.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/fce6dc21-8290-4444-ad41-6f6d113affab_isrForCrosstalkSources_1031703_56.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash a2accdbc85c3c9a6622c1e5307840fa4 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ddcb73e5-b00f-4de0-a2dc-fd5743b892d6_isrForCrosstalkSources_1031713_6.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ddcb73e5-b00f-4de0-a2dc-fd5743b892d6_isrForCrosstalkSources_1031713_6.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash a14268a5c83a8fb92b7490124b7c8cc1 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cee2b533-a441-4b29-bac7-a199759e4892_isrForCrosstalkSources_1031716_24.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cee2b533-a441-4b29-bac7-a199759e4892_isrForCrosstalkSources_1031716_24.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 622b583a1d5a700950b173147d7c0072 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7698a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fb91c96f-9fab-4107-b950-f50b4680799c_isrForCrosstalkSources_1031702_3.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fb91c96f-9fab-4107-b950-f50b4680799c_isrForCrosstalkSources_1031702_3.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 3236da852ce4b6871f61807c72c79adb parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/74c916ed-9d80-4035-9464-683a551f8052_isrForCrosstalkSources_1031737_51.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/74c916ed-9d80-4035-9464-683a551f8052_isrForCrosstalkSources_1031737_51.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 7c2a4c70db9521b368c665ecd5a4804e parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/496ef789-a5c0-46e2-b10c-ee15555d8af8_isrForCrosstalkSources_1031732_54.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/496ef789-a5c0-46e2-b10c-ee15555d8af8_isrForCrosstalkSources_1031732_54.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 8a48684776306d673338f0c428d45a4c parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/1d710df2-71d3-4716-9d89-9e8e3e9b296b_isrForCrosstalkSources_1031716_32.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/1d710df2-71d3-4716-9d89-9e8e3e9b296b_isrForCrosstalkSources_1031716_32.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 739aa4c6b3ac44ef364bf204d8c67a81 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7680e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/66b046f1-b983-44e3-8d6a-cc4d816bdaf2_isrForCrosstalkSources_1031715_24.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/66b046f1-b983-44e3-8d6a-cc4d816bdaf2_isrForCrosstalkSources_1031715_24.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 1ed0d350001831cfe7755603ce86a9a5 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/379e3d6b-bea4-414e-aac7-a7b39c03f7a7_isrForCrosstalkSources_1031738_31.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/379e3d6b-bea4-414e-aac7-a7b39c03f7a7_isrForCrosstalkSources_1031738_31.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 6212f50a59fe74eb193fd00b0df787b7 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7682c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/63837bd4-4315-47cf-bcda-827bb5c3839b_isrForCrosstalkSources_1031731_18.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/63837bd4-4315-47cf-bcda-827bb5c3839b_isrForCrosstalkSources_1031731_18.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 746d0a9b59fd561b10ea39ab28782d52 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/bfcc5f02-8124-4815-8fc4-4ec077c08156_isrForCrosstalkSources_1031726_21.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/bfcc5f02-8124-4815-8fc4-4ec077c08156_isrForCrosstalkSources_1031726_21.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 307b5f14f04b05077050eabbcf00e713 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a79b5650-d0d7-4671-8bc2-2902565b4e7d_isrForCrosstalkSources_1031721_51.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a79b5650-d0d7-4671-8bc2-2902565b4e7d_isrForCrosstalkSources_1031721_51.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash dc683686fb762c34d57e9efc29772426 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7684a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/aef745b4-48d1-46cc-8c2c-aaa2d115eba4_isrForCrosstalkSources_1031720_29.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/aef745b4-48d1-46cc-8c2c-aaa2d115eba4_isrForCrosstalkSources_1031720_29.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 9f2abfeca08626b73f237de1241ee19c parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7685e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e54e5775-45a6-4a10-a30d-3499abee317b_isrForCrosstalkSources_1031712_58.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e54e5775-45a6-4a10-a30d-3499abee317b_isrForCrosstalkSources_1031712_58.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash f5c8f029fdda42d6a5e04783515ce26d parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/fb21f9ff-918d-46a2-8ebf-1e62496a2cf1_isrForCrosstalkSources_1031708_18.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/fb21f9ff-918d-46a2-8ebf-1e62496a2cf1_isrForCrosstalkSources_1031708_18.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash a097f6a091aaf7b1ffc1ba121a41ae9f parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4133ad94-6d03-491c-9ae7-f6cf77b01950_isrForCrosstalkSources_1031709_12.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4133ad94-6d03-491c-9ae7-f6cf77b01950_isrForCrosstalkSources_1031709_12.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 214beafbf1a6a97112fa1b58eb31f4d9 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/5b5b9e2c-6fa6-4832-b01f-3cc93d86e09a_isrForCrosstalkSources_1031729_41.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/5b5b9e2c-6fa6-4832-b01f-3cc93d86e09a_isrForCrosstalkSources_1031729_41.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 4aa0ee2f1679572c0de910e6a62e949f parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/78af5620-ea33-4ecf-b8b4-36666604ebab_isrForCrosstalkSources_1031718_57.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/78af5620-ea33-4ecf-b8b4-36666604ebab_isrForCrosstalkSources_1031718_57.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 1e7baa47054ff053d21c92ac1b8817ef parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7687c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/c66dadcd-fe6d-4363-8a1e-eec49b8b26fe_isrForCrosstalkSources_1031721_3.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/c66dadcd-fe6d-4363-8a1e-eec49b8b26fe_isrForCrosstalkSources_1031721_3.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 15edae87ed83fa21d2bb2254199e2196 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/d07667de-d731-45be-b5c5-578b39569423_isrForCrosstalkSources_1031708_38.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/d07667de-d731-45be-b5c5-578b39569423_isrForCrosstalkSources_1031708_38.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 8a305c6f1a09ffd8c4765d9d7650b273 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/17806f93-ccca-463d-84fc-8b0a931d505c_isrForCrosstalkSources_1031728_13.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/17806f93-ccca-463d-84fc-8b0a931d505c_isrForCrosstalkSources_1031728_13.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 784a148fcf9d31c73ca172e2e3cd4960 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/66ed846e-dbec-4454-b085-d84fdefe1f8f_isrForCrosstalkSources_1031714_50.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/66ed846e-dbec-4454-b085-d84fdefe1f8f_isrForCrosstalkSources_1031714_50.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 063dae4f662e5861a21c2b6f9c2c4db0 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7689a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cd37dbe4-9417-4bfd-a699-3b92811f4230_isrForCrosstalkSources_1031707_5.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cd37dbe4-9417-4bfd-a699-3b92811f4230_isrForCrosstalkSources_1031707_5.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 8d1f402c8cf31eba922480150278020c parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/499e9489-f425-4e72-8ef9-f7540f950eaa_isrForCrosstalkSources_1031703_4.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/499e9489-f425-4e72-8ef9-f7540f950eaa_isrForCrosstalkSources_1031703_4.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 3417f2654e9fe4321ec27a8c753086ae parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2f23ba89-11a2-48fc-bb43-175fd0691a61_isrForCrosstalkSources_1031726_12.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2f23ba89-11a2-48fc-bb43-175fd0691a61_isrForCrosstalkSources_1031726_12.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 11e517559fa6439d0e5c6479baac7bb1 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/82e4b1be-f989-4bc5-8556-d0cc065ca407_isrForCrosstalkSources_1031711_20.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/82e4b1be-f989-4bc5-8556-d0cc065ca407_isrForCrosstalkSources_1031711_20.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 388d6fa564fdf698c2af4ae26b7e20b9 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8d784055-7008-453e-88b6-b5479ac1583b_isrForCrosstalkSources_1031732_11.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8d784055-7008-453e-88b6-b5479ac1583b_isrForCrosstalkSources_1031732_11.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash c80f0b9627198bec12e18b3ad13e67e8 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/9767639d-95f1-41ef-9e22-779ba07df2b0_isrForCrosstalkSources_1031705_46.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/9767639d-95f1-41ef-9e22-779ba07df2b0_isrForCrosstalkSources_1031705_46.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash bd7d0efacd98b968729dd30f1926ee75 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/aee614a6-2f70-4c0b-8b56-c05d5fdfc1cf_isrForCrosstalkSources_1031712_56.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/aee614a6-2f70-4c0b-8b56-c05d5fdfc1cf_isrForCrosstalkSources_1031712_56.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash adcc27788c56fe26a88d618743e3fcf8 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5d16131a-64ec-4fd9-a2b7-b02c4a28a374_isrForCrosstalkSources_1031735_39.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5d16131a-64ec-4fd9-a2b7-b02c4a28a374_isrForCrosstalkSources_1031735_39.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 4396d981e3b075070cd9f45c37ecedad parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/6a8f6aac-361e-44dd-9b74-fb4bcd6a5b51_isrForCrosstalkSources_1031704_45.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/6a8f6aac-361e-44dd-9b74-fb4bcd6a5b51_isrForCrosstalkSources_1031704_45.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash cd3afb9c9f86f9eb43733839848be335 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/fda1bbfb-d0cf-48ab-a7ce-fabbde48dcc2_isrForCrosstalkSources_1031718_9.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/fda1bbfb-d0cf-48ab-a7ce-fabbde48dcc2_isrForCrosstalkSources_1031718_9.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash b31342e98114c963b6cdb670c34a3a0c parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7691c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/22ab7c7d-7251-4e56-8f3e-628e43be0fa2_isrForCrosstalkSources_1031713_59.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/22ab7c7d-7251-4e56-8f3e-628e43be0fa2_isrForCrosstalkSources_1031713_59.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash d3a01be9bd018afa86d62bcf4fcf97fa parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/55526ad2-26a0-4332-80ae-6c800a5588b8_isrForCrosstalkSources_1031717_38.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/55526ad2-26a0-4332-80ae-6c800a5588b8_isrForCrosstalkSources_1031717_38.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash bf46981e643ad4c8391c28da7b90dbfc parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e768f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/265f60f9-4c48-4416-9e0d-f0272bb7e1a4_isrForCrosstalkSources_1031702_33.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/265f60f9-4c48-4416-9e0d-f0272bb7e1a4_isrForCrosstalkSources_1031702_33.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash c20572bff195fa3946f829247ea7df25 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ed710321-816b-4ddb-9882-6cd19c42d352_isrForCrosstalkSources_1031724_3.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ed710321-816b-4ddb-9882-6cd19c42d352_isrForCrosstalkSources_1031724_3.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 8873eb9a0659f20061b06899cd0b095a parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/0f4ecfa4-866b-480d-a782-2180f83f41fc_isrForCrosstalkSources_1031735_8.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/0f4ecfa4-866b-480d-a782-2180f83f41fc_isrForCrosstalkSources_1031735_8.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 3600e8c07d15030eaeeba4b9b7aad61c parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7694e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/62fa2a40-d9b7-4de4-a1fb-70230e781331_isrForCrosstalkSources_1031700_36.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/62fa2a40-d9b7-4de4-a1fb-70230e781331_isrForCrosstalkSources_1031700_36.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 7c3d999be628aba583dd1757ead600ae parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/840734b7-5f13-4aad-97d5-651117dd2ce1_isrForCrosstalkSources_1031699_32.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/840734b7-5f13-4aad-97d5-651117dd2ce1_isrForCrosstalkSources_1031699_32.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 0f04358dbc142da2fdb61fc7993d9ebb parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7693a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/18d1defe-f7cd-4d2f-a53d-9e572664f8ef_isrForCrosstalkSources_1031717_34.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/18d1defe-f7cd-4d2f-a53d-9e572664f8ef_isrForCrosstalkSources_1031717_34.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 558031a7e7a4cbfdbaf654c4e6d8b24d parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7696c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b856bf29-a283-470d-84a4-68e0f19aa9d9_isrForCrosstalkSources_1031707_47.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b856bf29-a283-470d-84a4-68e0f19aa9d9_isrForCrosstalkSources_1031707_47.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 5bff9b96507bfcc3ad0b370a6d0a0061 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/978b7c78-9e20-4072-ae3f-3e5485915ebf_isrForCrosstalkSources_1031704_35.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/978b7c78-9e20-4072-ae3f-3e5485915ebf_isrForCrosstalkSources_1031704_35.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash bd80ae7cb64995e5a65e27aaa9354482 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/1be21c15-6496-422a-9f59-77714aa59121_isrForCrosstalkSources_1031718_26.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/1be21c15-6496-422a-9f59-77714aa59121_isrForCrosstalkSources_1031718_26.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 60fff2e0ba407e18758a2cc3f3698b09 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a00ac069-754b-4ce9-b22f-8244985c5b09_isrForCrosstalkSources_1031712_24.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a00ac069-754b-4ce9-b22f-8244985c5b09_isrForCrosstalkSources_1031712_24.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 9d60afbe0dbf317ec556f9c5136c0ede parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/147b730b-556c-40a2-9245-88fed9c43d65_isrForCrosstalkSources_1031705_16.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/147b730b-556c-40a2-9245-88fed9c43d65_isrForCrosstalkSources_1031705_16.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 676de10495b069eb5291ef6d4abce6c7 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e7699e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/36cad2b1-3155-4eed-b8c0-149e55d08b07_isrForCrosstalkSources_1031727_17.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/36cad2b1-3155-4eed-b8c0-149e55d08b07_isrForCrosstalkSources_1031727_17.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 6e906ff1b257d0d67986b4362320dac6 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c8239c0f-46c0-47e4-84fb-d07edf4f032d_isrForCrosstalkSources_1031719_41.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c8239c0f-46c0-47e4-84fb-d07edf4f032d_isrForCrosstalkSources_1031719_41.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash b75c0aca0c68928bd41f5dc381efeedd parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/95f50193-d744-4e66-b3fb-a0bb541baf2d_isrForCrosstalkSources_1031721_11.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/95f50193-d744-4e66-b3fb-a0bb541baf2d_isrForCrosstalkSources_1031721_11.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 782e5af672c96d449a664c18eb7c8ab6 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/b45a8b6a-ac73-4c60-8b8a-bf5dcb745f59_isrForCrosstalkSources_1031729_49.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/b45a8b6a-ac73-4c60-8b8a-bf5dcb745f59_isrForCrosstalkSources_1031729_49.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 47817507f3c8590597ee7ddea91e7b66 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/9f357df4-c8fb-4a8c-aef7-9e485eae0dfd_isrForCrosstalkSources_1031720_50.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/9f357df4-c8fb-4a8c-aef7-9e485eae0dfd_isrForCrosstalkSources_1031720_50.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 3888f37c0db4f6b47faef126e4498c65 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/fea45413-0a38-4881-8dce-e8fd19813e40_isrForCrosstalkSources_1031733_52.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/fea45413-0a38-4881-8dce-e8fd19813e40_isrForCrosstalkSources_1031733_52.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash f123f00a9ffcfbd584c0a4c430d1b75a parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c7e7fba6-15bb-4149-a342-b885d2a7f811_isrForCrosstalkSources_1031725_51.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c7e7fba6-15bb-4149-a342-b885d2a7f811_isrForCrosstalkSources_1031725_51.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 6dedcab6ccf2c49c1638a8c29177ff50 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0c7cf628-924a-4efe-80a7-e56a41b70a9e_isrForCrosstalkSources_1031721_4.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0c7cf628-924a-4efe-80a7-e56a41b70a9e_isrForCrosstalkSources_1031721_4.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 7b199a3e419ad93eb481e9d5d3f56255 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/70d36247-dad6-4f75-8b63-129ac42c86ce_isrForCrosstalkSources_1031702_60.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/70d36247-dad6-4f75-8b63-129ac42c86ce_isrForCrosstalkSources_1031702_60.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 7ed465544de0d45d9d24a13f22246bdf parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/06fe5dfa-f6db-4597-961c-72ecb273d5db_isrForCrosstalkSources_1031725_55.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/06fe5dfa-f6db-4597-961c-72ecb273d5db_isrForCrosstalkSources_1031725_55.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 69f4bd0531d1a50c7d4f7d41d53c2d96 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/38cc467d-e139-4575-9c73-88b0ea2836e6_isrForCrosstalkSources_1031726_41.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/38cc467d-e139-4575-9c73-88b0ea2836e6_isrForCrosstalkSources_1031726_41.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash c35c76993bd1d68ccc5246a69bbcef7f parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/763fb7ea-ed18-40e8-a164-6881a3557b41_isrForCrosstalkSources_1031707_1.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/763fb7ea-ed18-40e8-a164-6881a3557b41_isrForCrosstalkSources_1031707_1.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 90ba0ce2763e975cdc2f65a9b731bbef parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/ad1c3320-25ad-4e0c-abae-f73305f13216_isrForCrosstalkSources_1031701_48.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/ad1c3320-25ad-4e0c-abae-f73305f13216_isrForCrosstalkSources_1031701_48.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash faf265ce8cb2ed9f13c1c3dc4f2abbcf parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/40d109fa-2d05-4118-b1dd-a4da6a9779d7_isrForCrosstalkSources_1031729_18.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/40d109fa-2d05-4118-b1dd-a4da6a9779d7_isrForCrosstalkSources_1031729_18.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 960d8080fc05fa68370baf7617efabef parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/bcde156b-b66b-4b95-9f49-0dd11ec945d3_isrForCrosstalkSources_1031705_52.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/bcde156b-b66b-4b95-9f49-0dd11ec945d3_isrForCrosstalkSources_1031705_52.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 8b42424197bb5b5ce9acf9f04b90acec parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/26f4215e-a5fe-4dbc-9261-4e0e68f3d4c6_isrForCrosstalkSources_1031711_41.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/26f4215e-a5fe-4dbc-9261-4e0e68f3d4c6_isrForCrosstalkSources_1031711_41.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 5baa63a8d7336d6e45f58946a20afdfd parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/90841de7-d469-4caf-bb34-56f6f024bd22_isrForCrosstalkSources_1031726_23.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/90841de7-d469-4caf-bb34-56f6f024bd22_isrForCrosstalkSources_1031726_23.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash d403ec491ea87723f293c58922bb5530 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d240881f-db7d-4fa9-be60-8ce92789a542_isrForCrosstalkSources_1031731_10.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d240881f-db7d-4fa9-be60-8ce92789a542_isrForCrosstalkSources_1031731_10.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash d5e777dd02085281974625a3520ff140 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:01:59 proc_lsst.multi:170 [INFO] [multi] job_id 17147931 proc_lsst.multi INFO: [multi] job_id 17147931 2024-03-17 04:01:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:01:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:01:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147931 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147931 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17147931 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-17 04:01:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-17 04:01:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:01:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:01:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:01:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:01:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:01:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:01:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0493b1cc-0185-4915-9835-d2f1c04f9a16_isrForCrosstalkSources_1031719_38.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0493b1cc-0185-4915-9835-d2f1c04f9a16_isrForCrosstalkSources_1031719_38.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash f413438fc156daabd17d4936eefaaa58 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/69509a94-a9b4-4dfa-b297-fd36b142d2d9_isrForCrosstalkSources_1031720_21.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/69509a94-a9b4-4dfa-b297-fd36b142d2d9_isrForCrosstalkSources_1031720_21.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash bd1de0e66802aff46d5b8222f78a9430 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/45cddbe6-218c-4f98-bb2d-173d051225f3_isrForCrosstalkSources_1031720_38.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/45cddbe6-218c-4f98-bb2d-173d051225f3_isrForCrosstalkSources_1031720_38.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 94f560152b5da69239096d07c634e50b parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e769760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8d919904-c040-4c57-9d8f-dbae865cffeb_isrForCrosstalkSources_1031735_19.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8d919904-c040-4c57-9d8f-dbae865cffeb_isrForCrosstalkSources_1031735_19.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash f9d180f5afe2504b5418ae41f396b9c4 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/742e2c8d-be80-4438-af53-52e2ccd860ee_isrForCrosstalkSources_1031729_36.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/742e2c8d-be80-4438-af53-52e2ccd860ee_isrForCrosstalkSources_1031729_36.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 5de65302ab5468141b3482f2cbd26074 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c724b541-fa99-4678-a1de-1a23da8f4edb_isrForCrosstalkSources_1031726_22.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c724b541-fa99-4678-a1de-1a23da8f4edb_isrForCrosstalkSources_1031726_22.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 75f940234f3ec3df12c7349eee62ce8d parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/cb32eddc-718b-488c-8f17-db9183c25e3c_isrForCrosstalkSources_1031718_52.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/cb32eddc-718b-488c-8f17-db9183c25e3c_isrForCrosstalkSources_1031718_52.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 71b5900736f282e5c24d1b5b43562ac5 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/632d85e8-72dd-4183-b922-4c8d9395b69e_isrForCrosstalkSources_1031711_27.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/632d85e8-72dd-4183-b922-4c8d9395b69e_isrForCrosstalkSources_1031711_27.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 82df5ed4242b9f9511588d7d5448ff67 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1598e231-3ce7-40e5-b5a5-b62d8eb29ba0_isrForCrosstalkSources_1031732_18.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1598e231-3ce7-40e5-b5a5-b62d8eb29ba0_isrForCrosstalkSources_1031732_18.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 25d5e143f6d2ed2a452deeed4f7997f5 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ed41865d-fbff-439f-a906-98bc807b45a0_isrForCrosstalkSources_1031715_51.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ed41865d-fbff-439f-a906-98bc807b45a0_isrForCrosstalkSources_1031715_51.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 6ff646f2e00756377b7e58985ada297f parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/d2c0b056-fa64-4db6-aa28-22cf31fa54ef_isrForCrosstalkSources_1031724_31.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/d2c0b056-fa64-4db6-aa28-22cf31fa54ef_isrForCrosstalkSources_1031724_31.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 74d894e490e23f9a6a766c9ad88096cc parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f26ae15e-fe6a-4176-a370-56e9f6d0f6e9_isrForCrosstalkSources_1031720_40.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f26ae15e-fe6a-4176-a370-56e9f6d0f6e9_isrForCrosstalkSources_1031720_40.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash eb876b4ac0e93da49d090227c96a3251 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/0b5f6a56-9426-4bdb-8c9f-3642dd1543fb_isrForCrosstalkSources_1031726_58.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/0b5f6a56-9426-4bdb-8c9f-3642dd1543fb_isrForCrosstalkSources_1031726_58.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash aba818cf0f0085928c3254db53f50611 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/eb1608a1-98d6-446a-babb-ba83d46e9f4f_isrForCrosstalkSources_1031718_15.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/eb1608a1-98d6-446a-babb-ba83d46e9f4f_isrForCrosstalkSources_1031718_15.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 877ddcd28cc59f950400516547dc2892 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/585116bd-37be-41ee-bbf4-cbba6bca9355_isrForCrosstalkSources_1031717_5.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/585116bd-37be-41ee-bbf4-cbba6bca9355_isrForCrosstalkSources_1031717_5.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 7b90bf386272aa6bffcc757480706c7a parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3949d8af-61f0-4d8a-98ab-2849af551681_isrForCrosstalkSources_1031738_22.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3949d8af-61f0-4d8a-98ab-2849af551681_isrForCrosstalkSources_1031738_22.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash eba3cd251cc021423566f0981baa762d parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4f4eb94e-ae95-4654-93f7-93f45824f02e_isrForCrosstalkSources_1031716_38.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4f4eb94e-ae95-4654-93f7-93f45824f02e_isrForCrosstalkSources_1031716_38.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash ac254c8c6f25585e0cf518afef2e71fc parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/256e1196-44aa-4eff-90d2-a3ed8a18eae2_isrForCrosstalkSources_1031703_37.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/256e1196-44aa-4eff-90d2-a3ed8a18eae2_isrForCrosstalkSources_1031703_37.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 04c32ea1db0d87b15babcf6a7c6d8ea1 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0b4ac318-fc9f-4bbc-a92f-0b5a581c2501_isrForCrosstalkSources_1031712_23.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0b4ac318-fc9f-4bbc-a92f-0b5a581c2501_isrForCrosstalkSources_1031712_23.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash c6983761576a26f242b6a3faf8b02ecf parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/554b4c99-c73a-46e7-8958-188c8c1d0b9a_isrForCrosstalkSources_1031703_17.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/554b4c99-c73a-46e7-8958-188c8c1d0b9a_isrForCrosstalkSources_1031703_17.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 210c805db24101936072794f01baaef2 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/1e56dedd-2a7c-4aad-93c1-3a55e5b72d78_isrForCrosstalkSources_1031738_46.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/1e56dedd-2a7c-4aad-93c1-3a55e5b72d78_isrForCrosstalkSources_1031738_46.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash ed6bc2fef0dfa3d15c2c58e7903fec2b parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2a90931b-0535-4973-8d66-767f30f07748_isrForCrosstalkSources_1031708_57.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2a90931b-0535-4973-8d66-767f30f07748_isrForCrosstalkSources_1031708_57.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 0f1456992ac402f7ec13ee63019d43ab parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3f9608cc-bc00-43b1-8a88-8c573b0dc553_isrForCrosstalkSources_1031707_49.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3f9608cc-bc00-43b1-8a88-8c573b0dc553_isrForCrosstalkSources_1031707_49.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash f40c13097e2d54636b52dac7197eab20 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/07c0b2ae-86d5-4390-90c5-4db883f5e670_isrForCrosstalkSources_1031729_56.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/07c0b2ae-86d5-4390-90c5-4db883f5e670_isrForCrosstalkSources_1031729_56.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 58807f3fb89895f36b47296112425642 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ff67b906-36e7-4f6e-97b6-cf9cc24b8743_isrForCrosstalkSources_1031729_55.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ff67b906-36e7-4f6e-97b6-cf9cc24b8743_isrForCrosstalkSources_1031729_55.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 127b430a21bb7ac64798a182a2d5afd4 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/84680fa7-8732-4e1a-9df7-7e84fc6e9202_isrForCrosstalkSources_1031738_23.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/84680fa7-8732-4e1a-9df7-7e84fc6e9202_isrForCrosstalkSources_1031738_23.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash fc923142657a183f6dfc7055a2d8d4b4 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0e4357f5-0154-4096-bd6d-166e81b0fe43_isrForCrosstalkSources_1031706_62.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0e4357f5-0154-4096-bd6d-166e81b0fe43_isrForCrosstalkSources_1031706_62.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 2133946618dba3e4447afe243ef09cb7 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/17d7ff65-0ae6-4176-a1c5-4a3bce7e3743_isrForCrosstalkSources_1031705_11.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/17d7ff65-0ae6-4176-a1c5-4a3bce7e3743_isrForCrosstalkSources_1031705_11.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 9a73cecd12f5b277355775c046b305f4 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/0fbf0f12-18a0-4988-82d3-7d51ba8b96b8_isrForCrosstalkSources_1031701_41.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/0fbf0f12-18a0-4988-82d3-7d51ba8b96b8_isrForCrosstalkSources_1031701_41.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash b72e3c50b37a5f520a174d6a506efd85 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ab980b2c-16f4-4ed3-852f-987234150c3b_isrForCrosstalkSources_1031729_60.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ab980b2c-16f4-4ed3-852f-987234150c3b_isrForCrosstalkSources_1031729_60.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 2b60aae93827d9c6cee4844bda01413c parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/4e4b4b4d-ca53-46eb-a0d7-b74132438b92_isrForCrosstalkSources_1031727_30.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/4e4b4b4d-ca53-46eb-a0d7-b74132438b92_isrForCrosstalkSources_1031727_30.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash c335d3634eeb13a1b32947a6962d110c parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/df4bdcd9-f35c-4336-b5b8-ccff721f2916_isrForCrosstalkSources_1031713_20.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/df4bdcd9-f35c-4336-b5b8-ccff721f2916_isrForCrosstalkSources_1031713_20.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 69204982ac1e27db4c27da6df8c1dac7 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/64306a60-cb9d-478b-9b46-c22facb3a691_isrForCrosstalkSources_1031706_58.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/64306a60-cb9d-478b-9b46-c22facb3a691_isrForCrosstalkSources_1031706_58.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 12ef5440976ea40d0b46e2b8fceb6582 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/f66bc53e-f9db-465d-b94b-a4ecd7f7afdd_isrForCrosstalkSources_1031719_48.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/f66bc53e-f9db-465d-b94b-a4ecd7f7afdd_isrForCrosstalkSources_1031719_48.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash f9d1b2f344cb10b838e306661f899145 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/bb6fc07b-fde2-4d1a-bfbe-8f258c001a06_isrForCrosstalkSources_1031708_20.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/bb6fc07b-fde2-4d1a-bfbe-8f258c001a06_isrForCrosstalkSources_1031708_20.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash f6552cdda8a1e40a91de6a74459d8d2b parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/a5942f87-2957-4fd9-b717-e6b39462a414_isrForCrosstalkSources_1031715_39.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/a5942f87-2957-4fd9-b717-e6b39462a414_isrForCrosstalkSources_1031715_39.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 91db99a10e6d24d84201033316e37d60 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8ceb9b0b-06f5-4018-b679-d030f48a0227_isrForCrosstalkSources_1031704_43.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8ceb9b0b-06f5-4018-b679-d030f48a0227_isrForCrosstalkSources_1031704_43.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 544cdae9ef2a9da59039f06239970015 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/4c0e8844-7641-4c35-adc0-920baa692150_isrForCrosstalkSources_1031734_34.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/4c0e8844-7641-4c35-adc0-920baa692150_isrForCrosstalkSources_1031734_34.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash ff13c1d809fd473530d295cd349f25f2 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/61cd904b-1275-432f-b592-44a3c831497b_isrForCrosstalkSources_1031714_10.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/61cd904b-1275-432f-b592-44a3c831497b_isrForCrosstalkSources_1031714_10.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 05b511ca50f854e8e073a8107c8128ae parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/25039d19-68a7-472a-bfcc-824a93c469b0_isrForCrosstalkSources_1031724_28.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/25039d19-68a7-472a-bfcc-824a93c469b0_isrForCrosstalkSources_1031724_28.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash c36139506b694b0cbb5be4f2d64acda5 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0f934b63-1950-4fcf-b605-fabe54658492_isrForCrosstalkSources_1031720_46.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0f934b63-1950-4fcf-b605-fabe54658492_isrForCrosstalkSources_1031720_46.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 3756f8e07b0dd11ef8368eaa63a45ca2 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/10b16b26-18c1-48e3-bdc8-987f07e2dbf9_isrForCrosstalkSources_1031714_27.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/10b16b26-18c1-48e3-bdc8-987f07e2dbf9_isrForCrosstalkSources_1031714_27.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash e41d7a5e2eeb0cf55fd26b43ac936260 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/697f3465-4a7c-4c8a-8fd5-406128457ebb_isrForCrosstalkSources_1031731_34.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/697f3465-4a7c-4c8a-8fd5-406128457ebb_isrForCrosstalkSources_1031731_34.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash bb9aaa7997c8fb10a831e42943dc96ac parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:01:59 proc_lsst.multi:170 [INFO] [multi] job_id 17147932 proc_lsst.multi INFO: [multi] job_id 17147932 2024-03-17 04:01:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 04:01:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:01:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147932 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147932 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17147932 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-17 04:01:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-17 04:01:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:01:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:01:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:01:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:01:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 04:01:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:01:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/f69d4123-d46d-4946-90f4-b257795f18c4_isrForCrosstalkSources_1031725_34.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/f69d4123-d46d-4946-90f4-b257795f18c4_isrForCrosstalkSources_1031725_34.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 7f03dc3ec6dbef3c9b7f832d2c6d5b36 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c054566e-3b95-4d12-9417-859b39b1ba73_isrForCrosstalkSources_1031726_30.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c054566e-3b95-4d12-9417-859b39b1ba73_isrForCrosstalkSources_1031726_30.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 42f2f9a243e8381737c4da454cc7e57b parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/91fcf096-8b1a-4ecb-89fa-75b35cbdccb8_isrForCrosstalkSources_1031738_8.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/91fcf096-8b1a-4ecb-89fa-75b35cbdccb8_isrForCrosstalkSources_1031738_8.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash a22aae3d73e5f3515dc9e8ad57daec85 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/bffb086e-4664-410c-ad48-a31a83b8baa5_isrForCrosstalkSources_1031706_49.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/bffb086e-4664-410c-ad48-a31a83b8baa5_isrForCrosstalkSources_1031706_49.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 07a1ee57c1a20840d29897912014c242 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4a9f3117-7fe3-4cf1-b83a-142b0d81fb9b_isrForCrosstalkSources_1031699_54.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4a9f3117-7fe3-4cf1-b83a-142b0d81fb9b_isrForCrosstalkSources_1031699_54.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash cd042ed225a9969eb39eeb7ca0fc6fcc parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/20161884-7164-4c31-8bce-6ddc2d7461a8_isrForCrosstalkSources_1031708_46.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/20161884-7164-4c31-8bce-6ddc2d7461a8_isrForCrosstalkSources_1031708_46.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash c6535aa98a827adc0e50fd4b52e4d99f parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/600c5c00-68eb-4806-8b3b-f12a7a2d62ef_isrForCrosstalkSources_1031730_33.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/600c5c00-68eb-4806-8b3b-f12a7a2d62ef_isrForCrosstalkSources_1031730_33.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 1b59ca6a58cc056d9bd500107a5a6d0a parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/4fba5702-6669-4b33-9dd7-44c699b86990_isrForCrosstalkSources_1031714_34.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/4fba5702-6669-4b33-9dd7-44c699b86990_isrForCrosstalkSources_1031714_34.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash de6b7283b660346007c58d98a98ee967 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a75c7288-0928-42a7-b7bd-d9cc8b08adde_isrForCrosstalkSources_1031706_24.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a75c7288-0928-42a7-b7bd-d9cc8b08adde_isrForCrosstalkSources_1031706_24.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 09872b6d013e556aeaf45406440cde7f parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f4df130b-1940-49b6-8a56-fd1fda09f559_isrForCrosstalkSources_1031727_12.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f4df130b-1940-49b6-8a56-fd1fda09f559_isrForCrosstalkSources_1031727_12.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 13c977cb81bd4d045b13e2f0074107c0 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c0a79642-ad2e-4bb3-9598-83e75042c57e_isrForCrosstalkSources_1031738_17.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c0a79642-ad2e-4bb3-9598-83e75042c57e_isrForCrosstalkSources_1031738_17.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash c494dee00f6ccfed1bf2592c786386fd parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9c3bfeb4-07e1-4474-9424-b764f0781f86_isrForCrosstalkSources_1031718_41.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9c3bfeb4-07e1-4474-9424-b764f0781f86_isrForCrosstalkSources_1031718_41.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 4dcde98f14404c1eff5f48405a7810f3 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3b35df55-e4ec-4fa7-a63d-219c3c8a4607_isrForCrosstalkSources_1031735_41.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3b35df55-e4ec-4fa7-a63d-219c3c8a4607_isrForCrosstalkSources_1031735_41.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash f7dc9fd5982980a138585c582ea38582 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1b1afef2-9423-4d2b-8619-2ade05af79f3_isrForCrosstalkSources_1031706_2.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1b1afef2-9423-4d2b-8619-2ade05af79f3_isrForCrosstalkSources_1031706_2.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 1edbafd903c699673df2fe78de904f56 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6032e216-305e-4866-8ce3-777dc515000b_isrForCrosstalkSources_1031723_36.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6032e216-305e-4866-8ce3-777dc515000b_isrForCrosstalkSources_1031723_36.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash e715c2101b51263cedbc13c2318039d9 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bb7e27c5-9cfb-4d9c-8611-8c3b5fb05646_isrForCrosstalkSources_1031715_42.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bb7e27c5-9cfb-4d9c-8611-8c3b5fb05646_isrForCrosstalkSources_1031715_42.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 3d0a2e0c247683ee90fbaa9cc50ebdcc parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/9041cd90-f65e-4454-8d91-c7621237a094_isrForCrosstalkSources_1031719_5.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/9041cd90-f65e-4454-8d91-c7621237a094_isrForCrosstalkSources_1031719_5.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 91b7164e85780477ee015269f83e7076 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/cac6da4c-7f0a-477b-b018-1fc150b5274d_isrForCrosstalkSources_1031702_59.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/cac6da4c-7f0a-477b-b018-1fc150b5274d_isrForCrosstalkSources_1031702_59.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash d01f793b6caa4b155e02187eca9ad0dc parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7b69e4a0-11b7-4fe3-9491-5f791a1d4424_isrForCrosstalkSources_1031702_15.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7b69e4a0-11b7-4fe3-9491-5f791a1d4424_isrForCrosstalkSources_1031702_15.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash fe3f108e52890ce9943690d3d3aa51fa parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/4c1545db-8c72-454b-9110-3c911ea958f1_isrForCrosstalkSources_1031701_1.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/4c1545db-8c72-454b-9110-3c911ea958f1_isrForCrosstalkSources_1031701_1.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 539c09066a8bc65fb2dd52744a0b4997 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/338b2f46-a5a8-45a7-8dcd-6814b94a5616_isrForCrosstalkSources_1031715_28.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/338b2f46-a5a8-45a7-8dcd-6814b94a5616_isrForCrosstalkSources_1031715_28.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash da0d316b60b6110e78bb147f40612798 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7d591371-7e43-434c-bf0f-99f6240368e1_isrForCrosstalkSources_1031732_10.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7d591371-7e43-434c-bf0f-99f6240368e1_isrForCrosstalkSources_1031732_10.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 752d2fe2fa0be2a35b371972873eded4 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1139df6e-0d11-46b6-88a6-ee62ba724dae_isrForCrosstalkSources_1031713_62.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1139df6e-0d11-46b6-88a6-ee62ba724dae_isrForCrosstalkSources_1031713_62.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash fadee834033094b521b502a718cc916e parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a78ca919-e4c9-4324-bbdf-fa2956e5ea18_isrForCrosstalkSources_1031738_21.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a78ca919-e4c9-4324-bbdf-fa2956e5ea18_isrForCrosstalkSources_1031738_21.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash b6eaa04ac3c227fb79c3d311a275a796 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/87885509-2836-4cb0-8d54-92ba336f3dcc_isrForCrosstalkSources_1031736_11.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/87885509-2836-4cb0-8d54-92ba336f3dcc_isrForCrosstalkSources_1031736_11.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash f2d422d79ada674c6bded4829f539000 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a4c4d095-c222-46a9-9193-592a858aedff_isrForCrosstalkSources_1031731_28.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a4c4d095-c222-46a9-9193-592a858aedff_isrForCrosstalkSources_1031731_28.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash f9428c0ae1d36c5b6f31758203afb217 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/4399d1b4-2792-4a10-9494-df81ca5112df_isrForCrosstalkSources_1031734_54.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/4399d1b4-2792-4a10-9494-df81ca5112df_isrForCrosstalkSources_1031734_54.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash b8e268efcc50b5f3dcb938c8c32438ac parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d9c3f317-aaa9-44a3-a00b-b95d4f9b9e24_isrForCrosstalkSources_1031734_4.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d9c3f317-aaa9-44a3-a00b-b95d4f9b9e24_isrForCrosstalkSources_1031734_4.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash e02f8cc812dd500d042cb7186108955a parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/05605a98-32aa-495d-b39c-1498b00e7894_isrForCrosstalkSources_1031735_12.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/05605a98-32aa-495d-b39c-1498b00e7894_isrForCrosstalkSources_1031735_12.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash c5e5b950c8c1c52d0a8c03d6fb0085a8 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/dfe0ae5b-d690-41aa-be86-ab8cdd2fd5f7_isrForCrosstalkSources_1031729_30.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/dfe0ae5b-d690-41aa-be86-ab8cdd2fd5f7_isrForCrosstalkSources_1031729_30.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash cca9646743e9855cc65dec77f9fef23b parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/255e43a8-0a21-41c5-a7b5-913a6939b54f_isrForCrosstalkSources_1031735_23.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/255e43a8-0a21-41c5-a7b5-913a6939b54f_isrForCrosstalkSources_1031735_23.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 6e1f6f6d710a5a844d995ddc455b1658 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/39312ad2-42b0-45a2-98c3-420e7494f672_isrForCrosstalkSources_1031720_17.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/39312ad2-42b0-45a2-98c3-420e7494f672_isrForCrosstalkSources_1031720_17.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash b761654bc40eb0629629e6d5a89e62f6 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/86db2fba-551f-442c-abdd-195d2c57c210_isrForCrosstalkSources_1031733_56.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/86db2fba-551f-442c-abdd-195d2c57c210_isrForCrosstalkSources_1031733_56.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 23f5fd21b095ca3aa7a6926d8a98727b parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c450a3e6-f729-4993-9c60-ea632f1cd6ca_isrForCrosstalkSources_1031705_10.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c450a3e6-f729-4993-9c60-ea632f1cd6ca_isrForCrosstalkSources_1031705_10.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash a857bc6ca44b3ca5c58127b29782560f parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bc103cfb-1a2f-41db-ae7b-e2f2469f348d_isrForCrosstalkSources_1031734_29.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bc103cfb-1a2f-41db-ae7b-e2f2469f348d_isrForCrosstalkSources_1031734_29.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 6d59ad2f9f40e91a603333fcd6315b16 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/b1b95903-8407-4aff-a0b9-2ffb0d1c2b72_isrForCrosstalkSources_1031699_46.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/b1b95903-8407-4aff-a0b9-2ffb0d1c2b72_isrForCrosstalkSources_1031699_46.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash d6e6fa4396ee8e976a46b2f8c5833b2f parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3817f605-5510-483a-a68d-5f93548141ba_isrForCrosstalkSources_1031716_46.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3817f605-5510-483a-a68d-5f93548141ba_isrForCrosstalkSources_1031716_46.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 27c71b1da7f5929ebe1c53feddfd7d18 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/56f18e4e-1684-4d7e-ac17-fdce6ae3b795_isrForCrosstalkSources_1031729_52.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/56f18e4e-1684-4d7e-ac17-fdce6ae3b795_isrForCrosstalkSources_1031729_52.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 5128d60da76107af4d85e435181e7d97 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/e367b07b-e231-4aed-b32c-bdec41424c00_isrForCrosstalkSources_1031719_26.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/e367b07b-e231-4aed-b32c-bdec41424c00_isrForCrosstalkSources_1031719_26.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 92351f89f86c33471741014202ef9494 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67116442-5240-4ac0-8b23-99d1a0502d8b_isrForCrosstalkSources_1031728_29.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67116442-5240-4ac0-8b23-99d1a0502d8b_isrForCrosstalkSources_1031728_29.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 8401cb08a4e42b8f50192a30f7a14752 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b97ee244-a90b-4ef8-9e13-115434848bba_isrForCrosstalkSources_1031726_48.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b97ee244-a90b-4ef8-9e13-115434848bba_isrForCrosstalkSources_1031726_48.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 4d077f02271571e405e4a7e2706ee1c2 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/ba839e26-5a9d-46ac-aaaa-7166fde00e22_isrForCrosstalkSources_1031736_34.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/ba839e26-5a9d-46ac-aaaa-7166fde00e22_isrForCrosstalkSources_1031736_34.stderr 2024-03-17 04:02:00 proc_lsst.multi:170 [INFO] [multi] job_id 17147933 proc_lsst.multi INFO: [multi] job_id 17147933 2024-03-17 04:02:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 04:02:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147933 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147933 parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17147933 parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.executors.status_handling INFO: Allocated block ID 4 parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-17 04:02:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-17 04:02:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 04:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash fb431f49c0ce1e3d452cb8b30cab55b7 parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f10ba1c5-9892-4dc1-9c58-5dfd1978956f_isrForCrosstalkSources_1031700_60.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f10ba1c5-9892-4dc1-9c58-5dfd1978956f_isrForCrosstalkSources_1031700_60.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash a9137b5725ec6414f16e786cccc58fb9 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e76bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9afc5ac4-72ee-4f25-abe7-10e7c2590a65_isrForCrosstalkSources_1031729_45.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9afc5ac4-72ee-4f25-abe7-10e7c2590a65_isrForCrosstalkSources_1031729_45.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 67b3f5dfa5cb19b031e856329e91a969 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ebec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/decc39da-8a5a-4dcd-9999-d42fc74af517_isrForCrosstalkSources_1031718_18.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/decc39da-8a5a-4dcd-9999-d42fc74af517_isrForCrosstalkSources_1031718_18.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash bc4ccdb8a5bcbab429431a4b4b5fe2c5 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b7742569-a3ea-4422-8711-747fc79ba280_isrForCrosstalkSources_1031726_8.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b7742569-a3ea-4422-8711-747fc79ba280_isrForCrosstalkSources_1031726_8.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 5bb09de0dd7bb1bee58da75cab4765f5 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/33fddf2c-d184-42bc-9f0f-58b3753fcdd2_isrForCrosstalkSources_1031707_53.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/33fddf2c-d184-42bc-9f0f-58b3753fcdd2_isrForCrosstalkSources_1031707_53.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash c961c4a8442ffd5674825984442a97c6 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/73abf7de-1a7f-4995-a802-62d9f47176fc_isrForCrosstalkSources_1031709_44.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/73abf7de-1a7f-4995-a802-62d9f47176fc_isrForCrosstalkSources_1031709_44.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 38023e1475c9c427ef6553a9999a159e parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/8e54480a-e077-4954-8b59-e2645080f8ff_isrForCrosstalkSources_1031725_10.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/8e54480a-e077-4954-8b59-e2645080f8ff_isrForCrosstalkSources_1031725_10.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 49d3c88a9b5093e1f572e78b73aa91b3 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ca536097-50ae-4545-b89d-713e1ed3f966_isrForCrosstalkSources_1031713_29.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ca536097-50ae-4545-b89d-713e1ed3f966_isrForCrosstalkSources_1031713_29.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 702bbd3c48fb89725b411500ec6e6572 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d234ae18-87a8-4473-9727-7240336e9ffb_isrForCrosstalkSources_1031704_13.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d234ae18-87a8-4473-9727-7240336e9ffb_isrForCrosstalkSources_1031704_13.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 9b1eea3a21a2dd90ca6a0b48365b6fdf parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/01c4e492-0f4a-4d6c-9449-e25d8718b868_isrForCrosstalkSources_1031738_27.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/01c4e492-0f4a-4d6c-9449-e25d8718b868_isrForCrosstalkSources_1031738_27.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 1b9387d9ee5b301b908a420e9caa38b8 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/294a3296-b8d3-4386-b651-ce31ce5bc78e_isrForCrosstalkSources_1031710_6.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/294a3296-b8d3-4386-b651-ce31ce5bc78e_isrForCrosstalkSources_1031710_6.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 5a70431c1e10a260d01417a223abe60b parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/2831b89f-265b-434a-80ed-0251154ac974_isrForCrosstalkSources_1031700_53.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/2831b89f-265b-434a-80ed-0251154ac974_isrForCrosstalkSources_1031700_53.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash bac0e84df06be105f9ff3b35a60dbbe1 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/369b8bd8-2e7f-4a99-9830-7d13dc606efa_isrForCrosstalkSources_1031721_26.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/369b8bd8-2e7f-4a99-9830-7d13dc606efa_isrForCrosstalkSources_1031721_26.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 3c1948c8a9d5525f35e5a0c4eda7db3a parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f19537d0-12fe-402f-abef-b34aa317021c_isrForCrosstalkSources_1031735_1.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f19537d0-12fe-402f-abef-b34aa317021c_isrForCrosstalkSources_1031735_1.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 997d264cb6af3c5175c38db47177af26 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/88f6663c-1461-4f83-a07b-580eff8e4798_isrForCrosstalkSources_1031712_61.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/88f6663c-1461-4f83-a07b-580eff8e4798_isrForCrosstalkSources_1031712_61.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 27453974be202a17d13c9ecb28f2384b parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d9de3c61-5f5a-4894-88ba-59190823befe_isrForCrosstalkSources_1031734_5.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d9de3c61-5f5a-4894-88ba-59190823befe_isrForCrosstalkSources_1031734_5.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash df7fe2242bfaf2d08807e3ed1d303848 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/910d7746-f200-416a-a5aa-469ac87f3865_isrForCrosstalkSources_1031731_46.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/910d7746-f200-416a-a5aa-469ac87f3865_isrForCrosstalkSources_1031731_46.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 9f37773bf3eba710c4769f6be90f1378 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/0c333068-0a7a-4bdd-9c55-85d531d804f1_isrForCrosstalkSources_1031713_23.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/0c333068-0a7a-4bdd-9c55-85d531d804f1_isrForCrosstalkSources_1031713_23.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash e2e8c8369feb711d33128e2d8eeed162 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/b0a3bbde-75de-4db5-a62d-f207ccefb614_isrForCrosstalkSources_1031711_38.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/b0a3bbde-75de-4db5-a62d-f207ccefb614_isrForCrosstalkSources_1031711_38.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 58733ac8a8bb445fdf92ad1837156a2a parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/2aaca8b1-7f94-4f58-8ac6-5c5aee97a91e_isrForCrosstalkSources_1031706_3.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/2aaca8b1-7f94-4f58-8ac6-5c5aee97a91e_isrForCrosstalkSources_1031706_3.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 8d06dae6338ffe88557bdf5687a7560d parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/6c451e91-8877-46f9-b099-a6cd315e8c05_isrForCrosstalkSources_1031701_9.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/6c451e91-8877-46f9-b099-a6cd315e8c05_isrForCrosstalkSources_1031701_9.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 24723e3b547b25231cf686b1167fc154 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8653db51-d6b7-4921-a419-333ae67f3ac4_isrForCrosstalkSources_1031733_15.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8653db51-d6b7-4921-a419-333ae67f3ac4_isrForCrosstalkSources_1031733_15.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash cb320cb9ea535deb719f07f6fde0158b parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1050fc2d-1497-457e-aca5-6c298e0ff7f9_isrForCrosstalkSources_1031735_4.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1050fc2d-1497-457e-aca5-6c298e0ff7f9_isrForCrosstalkSources_1031735_4.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 2fb110f0c2d918e457e67bd1b2e99f9a parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/d89b32e8-ae99-43c1-aec9-7ca34eef05b8_isrForCrosstalkSources_1031723_22.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/d89b32e8-ae99-43c1-aec9-7ca34eef05b8_isrForCrosstalkSources_1031723_22.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 4e1d65ade7f868480b86df3714335808 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/38186904-e514-46ca-ad7d-d04c6d626d99_isrForCrosstalkSources_1031724_50.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/38186904-e514-46ca-ad7d-d04c6d626d99_isrForCrosstalkSources_1031724_50.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash bb39c39fefcef04d40aec3d1259c8f38 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f5685629-b750-43d8-af11-34ca8b26c4fa_isrForCrosstalkSources_1031732_25.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f5685629-b750-43d8-af11-34ca8b26c4fa_isrForCrosstalkSources_1031732_25.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 07db0967d667bac801aa6f976371ba14 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/0d26086f-0400-4357-a8ce-958e5445893b_isrForCrosstalkSources_1031709_60.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/0d26086f-0400-4357-a8ce-958e5445893b_isrForCrosstalkSources_1031709_60.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash a303340e641d937d1b5643ced26c875e parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/8e4cf9fa-aa36-4cd8-82a5-da99eee67d1b_isrForCrosstalkSources_1031734_61.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/8e4cf9fa-aa36-4cd8-82a5-da99eee67d1b_isrForCrosstalkSources_1031734_61.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 342d46511197c83ff66653336ba7e10a parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9f0f1a8b-33e2-4f7b-824e-ae991801080a_isrForCrosstalkSources_1031716_10.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9f0f1a8b-33e2-4f7b-824e-ae991801080a_isrForCrosstalkSources_1031716_10.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 5b0a43f767066d9605da5f3b110d110d parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/f1e08434-d101-4699-8b51-64ea93490f0a_isrForCrosstalkSources_1031707_59.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/f1e08434-d101-4699-8b51-64ea93490f0a_isrForCrosstalkSources_1031707_59.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 77adb2825086c086d7579eabdf9fa2c5 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5ef739d0-5422-4a8c-8490-57b0319fcf99_isrForCrosstalkSources_1031727_14.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5ef739d0-5422-4a8c-8490-57b0319fcf99_isrForCrosstalkSources_1031727_14.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash d0487716917a1959298109f2b7c8c378 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/eb05d2d2-9a7e-4c06-b4ab-c8700ce583a8_isrForCrosstalkSources_1031705_37.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/eb05d2d2-9a7e-4c06-b4ab-c8700ce583a8_isrForCrosstalkSources_1031705_37.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 7a3cb37f32ab3634a825a02cc984ee03 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ead40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2e4140a2-9980-4f0d-9c9f-3b834d92cba2_isrForCrosstalkSources_1031709_59.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2e4140a2-9980-4f0d-9c9f-3b834d92cba2_isrForCrosstalkSources_1031709_59.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 392715047b8a45a3088c71f18135ff79 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/45462955-3be3-4241-bce6-423110fad208_isrForCrosstalkSources_1031715_52.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/45462955-3be3-4241-bce6-423110fad208_isrForCrosstalkSources_1031715_52.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 53d7ed31aeded14b8b73ee54bcdd7c23 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/21c7f0c1-71ce-467d-bbab-5ffd69953038_isrForCrosstalkSources_1031723_53.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/21c7f0c1-71ce-467d-bbab-5ffd69953038_isrForCrosstalkSources_1031723_53.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 8a133386affa277c45f3199026a3b49b parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f07ae3c1-6343-4c11-9909-d6aa9922636a_isrForCrosstalkSources_1031716_58.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f07ae3c1-6343-4c11-9909-d6aa9922636a_isrForCrosstalkSources_1031716_58.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 19459dde110fe6bcc82791a123b1e7bc parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/1b1fd11a-8783-451b-86a7-7a0b2eec9b57_isrForCrosstalkSources_1031721_8.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/1b1fd11a-8783-451b-86a7-7a0b2eec9b57_isrForCrosstalkSources_1031721_8.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 6f8a0e4b310b1c203de801ca45ddc5eb parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c238c19d-0a6b-4f9b-a774-e389d31e072c_isrForCrosstalkSources_1031726_38.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c238c19d-0a6b-4f9b-a774-e389d31e072c_isrForCrosstalkSources_1031726_38.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 880031ecc7d9ed8598f287066b89a590 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/357c3619-7bd7-43f4-a298-52bc04bde48c_isrForCrosstalkSources_1031699_62.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/357c3619-7bd7-43f4-a298-52bc04bde48c_isrForCrosstalkSources_1031699_62.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash dc222cab1ebcfaa2fdee0769d453cacb parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2bb075b3-cede-4a14-9203-b4040d88d390_isrForCrosstalkSources_1031704_21.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2bb075b3-cede-4a14-9203-b4040d88d390_isrForCrosstalkSources_1031704_21.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash d3c39fba4e636e3adeccfa6d80841f90 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/e5e9cecd-3e34-4df6-8991-46a7dbbbb459_isrForCrosstalkSources_1031724_8.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/e5e9cecd-3e34-4df6-8991-46a7dbbbb459_isrForCrosstalkSources_1031724_8.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 7470cab877ddbd94e2c3263a4743209e parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/fff4e98d-1a6d-41fa-85ed-5901c678b436_isrForCrosstalkSources_1031704_40.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/fff4e98d-1a6d-41fa-85ed-5901c678b436_isrForCrosstalkSources_1031704_40.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 987b5fedfa44926de012dda6a145a0da parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5a41a18f-24ec-49f4-bd32-ca8de2c5b490_isrForCrosstalkSources_1031717_37.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5a41a18f-24ec-49f4-bd32-ca8de2c5b490_isrForCrosstalkSources_1031717_37.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash ea6e855047cc364e35ce1cef62bd21cc parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0187ac5d-050e-45ea-a6f8-2b425fe3b4fb_isrForCrosstalkSources_1031705_2.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0187ac5d-050e-45ea-a6f8-2b425fe3b4fb_isrForCrosstalkSources_1031705_2.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 8b623d9aaa6328108bb5b32594cabfd6 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5515bee1-eb47-42b1-816e-58fe233eeeb8_isrForCrosstalkSources_1031717_33.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5515bee1-eb47-42b1-816e-58fe233eeeb8_isrForCrosstalkSources_1031717_33.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash ef27b3b97039a8f554214dff92132b35 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/2516cc82-5c08-44d4-bddd-922f3e153dac_isrForCrosstalkSources_1031728_9.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/2516cc82-5c08-44d4-bddd-922f3e153dac_isrForCrosstalkSources_1031728_9.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash dc2175e071f7558c0f61c3c80f374436 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8eba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/669191a9-b24f-42c3-ae0b-58c567548e28_isrForCrosstalkSources_1031714_35.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/669191a9-b24f-42c3-ae0b-58c567548e28_isrForCrosstalkSources_1031714_35.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 17551e697c9459c497066ca5a2ebab57 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ebba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3c653a90-9c94-44fd-825c-ea3ec87e1c66_isrForCrosstalkSources_1031731_25.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3c653a90-9c94-44fd-825c-ea3ec87e1c66_isrForCrosstalkSources_1031731_25.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 81256a402016d3d9582e8554782b1718 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d81bb3a3-b19a-4964-86d2-396bc1ebd907_isrForCrosstalkSources_1031728_34.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d81bb3a3-b19a-4964-86d2-396bc1ebd907_isrForCrosstalkSources_1031728_34.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 109e515ac6ce98cd8bcf4be2750bcc17 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ebc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:01 proc_lsst.multi:170 [INFO] [multi] job_id 17147934 proc_lsst.multi INFO: [multi] job_id 17147934 2024-03-17 04:02:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 04:02:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147934 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147934 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17147934 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-17 04:02:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-17 04:02:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 04:02:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/84315f07-e141-40d9-a605-42692dbc3d76_isrForCrosstalkSources_1031699_26.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/84315f07-e141-40d9-a605-42692dbc3d76_isrForCrosstalkSources_1031699_26.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash f9fbc0cb28f7319b790f3063f4a3a5a9 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.providers.slurm.slurm DEBUG: moving files parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ebce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d9de4f7d-399c-4df5-8586-eab7af19b484_isrForCrosstalkSources_1031738_40.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d9de4f7d-399c-4df5-8586-eab7af19b484_isrForCrosstalkSources_1031738_40.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 4c7d8baec9519af9f399dbbef2cb079e parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a8812683-2f24-4e9a-b6e0-9dbb86ea4f69_isrForCrosstalkSources_1031720_31.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a8812683-2f24-4e9a-b6e0-9dbb86ea4f69_isrForCrosstalkSources_1031720_31.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 6fd77320f249818ef498ebcbf476303c parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ebf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/5ff06e0f-b27e-4ac4-bc20-bd2ccc90ce2c_isrForCrosstalkSources_1031719_4.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/5ff06e0f-b27e-4ac4-bc20-bd2ccc90ce2c_isrForCrosstalkSources_1031719_4.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash a603201de2f2322874afa8d4ba67796e parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ebd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a746de3c-11b4-4e95-8e8f-91056c854f95_isrForCrosstalkSources_1031713_49.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a746de3c-11b4-4e95-8e8f-91056c854f95_isrForCrosstalkSources_1031713_49.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash a55c279b23b71ca846230079d6dd524f parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ea020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/b78cda99-9249-4b6c-a3a7-a4d41aa5e8da_isrForCrosstalkSources_1031731_4.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/b78cda99-9249-4b6c-a3a7-a4d41aa5e8da_isrForCrosstalkSources_1031731_4.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 9c3139f0ff29a9a04d0bb380c2862e5e parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/128feb0c-c466-44e7-8d5b-79015596dd35_isrForCrosstalkSources_1031728_42.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/128feb0c-c466-44e7-8d5b-79015596dd35_isrForCrosstalkSources_1031728_42.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 7363317be69bb9eccee6a6a3ae9e6d7a parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/9d8f524c-596d-4c8a-a905-44a07ec7bffa_isrForCrosstalkSources_1031736_2.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/9d8f524c-596d-4c8a-a905-44a07ec7bffa_isrForCrosstalkSources_1031736_2.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 7bb73a41e937637be8f1ebe6eeccced2 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/34b0110c-c0e4-47a3-927e-03c39b0c674f_isrForCrosstalkSources_1031723_21.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/34b0110c-c0e4-47a3-927e-03c39b0c674f_isrForCrosstalkSources_1031723_21.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 7472244cd9b12d6756c762f2f88ac701 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/436a93d5-36ca-4598-aefb-085bef7f37a3_isrForCrosstalkSources_1031704_34.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/436a93d5-36ca-4598-aefb-085bef7f37a3_isrForCrosstalkSources_1031704_34.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 3836f24ccfad6348ee3c35cde2883a92 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d9b8ddfc-d4a6-4ea4-8e57-f2722409af1f_isrForCrosstalkSources_1031714_42.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d9b8ddfc-d4a6-4ea4-8e57-f2722409af1f_isrForCrosstalkSources_1031714_42.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 8549f0ab7d4f97347273b1c70a806bbb parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/45db14f5-f5c1-4df3-be92-937def6e74cd_isrForCrosstalkSources_1031704_4.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/45db14f5-f5c1-4df3-be92-937def6e74cd_isrForCrosstalkSources_1031704_4.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 5efe29d0e0ece89d7736f0eacc635374 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/e7f916df-fa99-43b6-aefb-77879d13268d_isrForCrosstalkSources_1031718_53.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/e7f916df-fa99-43b6-aefb-77879d13268d_isrForCrosstalkSources_1031718_53.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash a4609188318f72466d0ec262201c92ab parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ec9ff92c-c3bf-4465-9811-a8a5ef228117_isrForCrosstalkSources_1031715_27.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ec9ff92c-c3bf-4465-9811-a8a5ef228117_isrForCrosstalkSources_1031715_27.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 3c6b12bea92e73c5f19e0e06c4784d3a parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1e57ec1f-b209-45e5-a986-739aa777354e_isrForCrosstalkSources_1031706_28.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1e57ec1f-b209-45e5-a986-739aa777354e_isrForCrosstalkSources_1031706_28.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash b69b7b60f208b36838453600f0807eb8 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/b356972c-d1c5-4915-b6a1-3e18016a7bad_isrForCrosstalkSources_1031732_55.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/b356972c-d1c5-4915-b6a1-3e18016a7bad_isrForCrosstalkSources_1031732_55.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 4d281fa04c0501caa41c2717c6a232e2 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/36eae125-e68f-43d6-8809-619f7fcd45c7_isrForCrosstalkSources_1031707_60.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/36eae125-e68f-43d6-8809-619f7fcd45c7_isrForCrosstalkSources_1031707_60.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash ee4df680408b3448d826cfcc58c22402 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a6cf886d-b598-4d3a-8874-019bb9e57a95_isrForCrosstalkSources_1031722_47.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a6cf886d-b598-4d3a-8874-019bb9e57a95_isrForCrosstalkSources_1031722_47.stderr parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 3cd3b6312bc9fe75bc0e97d6586b7f0d parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1263c925-8207-47c0-a52a-970f30b5668a_isrForCrosstalkSources_1031711_23.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1263c925-8207-47c0-a52a-970f30b5668a_isrForCrosstalkSources_1031711_23.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 7cec234855416715113c64da010f7fc2 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9525813a-6289-458a-baed-72106b4af81b_isrForCrosstalkSources_1031730_32.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9525813a-6289-458a-baed-72106b4af81b_isrForCrosstalkSources_1031730_32.stderr parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash 82e253a6c8e9ee62ec3fadc30a251c16 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/acb4261d-e9f9-48bb-a925-da5d2d76de97_isrForCrosstalkSources_1031718_21.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/acb4261d-e9f9-48bb-a925-da5d2d76de97_isrForCrosstalkSources_1031718_21.stderr parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash b9b1992d884309b5ce567462c2bb2d4a parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a14f61d9-a7a6-475f-852b-e7783130497e_isrForCrosstalkSources_1031711_21.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a14f61d9-a7a6-475f-852b-e7783130497e_isrForCrosstalkSources_1031711_21.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 7b9641092cac1a5811b67b574793ac29 parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/27d4887c-ced7-4a44-8cc0-39377afdf279_isrForCrosstalkSources_1031708_53.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/27d4887c-ced7-4a44-8cc0-39377afdf279_isrForCrosstalkSources_1031708_53.stderr parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 83cb2df445981e8859032313adbd5f6a parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0e5c80d7-7466-4c2f-9be5-ae895c080328_isrForCrosstalkSources_1031721_13.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0e5c80d7-7466-4c2f-9be5-ae895c080328_isrForCrosstalkSources_1031721_13.stderr parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash ef346b30c782b104ab96a6068448ce38 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/553e1b15-2d0e-4048-9bac-ff20365e8250_isrForCrosstalkSources_1031704_54.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/553e1b15-2d0e-4048-9bac-ff20365e8250_isrForCrosstalkSources_1031704_54.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash e945f05d4a9f8c01763e32ebe64c8c13 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d6e34c3d-4a89-4e22-a905-ea6bc9a63364_isrForCrosstalkSources_1031730_60.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d6e34c3d-4a89-4e22-a905-ea6bc9a63364_isrForCrosstalkSources_1031730_60.stderr parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 8cf0968078bd561f2574c2e40e2c9ec6 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d6a125d4-686e-4ab8-af20-183d1900ceb5_isrForCrosstalkSources_1031734_1.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d6a125d4-686e-4ab8-af20-183d1900ceb5_isrForCrosstalkSources_1031734_1.stderr parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash be2859e939168450503bb8fbf685d437 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8693b0c3-b364-44b3-a0dc-b44efc9bbef2_isrForCrosstalkSources_1031700_6.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8693b0c3-b364-44b3-a0dc-b44efc9bbef2_isrForCrosstalkSources_1031700_6.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 2a701f7932ba07fcf5f8d622bd7aec60 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c2aa801b-7583-4e01-9538-d2e9be66747c_isrForCrosstalkSources_1031718_55.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c2aa801b-7583-4e01-9538-d2e9be66747c_isrForCrosstalkSources_1031718_55.stderr parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 8e5adf47f549474437a2a25298a68c76 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/65aa0be0-2f0f-4e6b-b432-1392b049714a_isrForCrosstalkSources_1031732_1.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/65aa0be0-2f0f-4e6b-b432-1392b049714a_isrForCrosstalkSources_1031732_1.stderr parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 0d48b4d4f97d751a6546e3d07e8172ef parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/5e2cce73-b6ca-4fe5-9665-ffde49f4a57d_isrForCrosstalkSources_1031718_38.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/5e2cce73-b6ca-4fe5-9665-ffde49f4a57d_isrForCrosstalkSources_1031718_38.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 7711b2205c7c2e1ad023571863014fc4 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/c1ed0e56-0706-45c1-8e0b-d4f4173a368b_isrForCrosstalkSources_1031736_44.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/c1ed0e56-0706-45c1-8e0b-d4f4173a368b_isrForCrosstalkSources_1031736_44.stderr parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 63ab57ee47f05cc8e4db6c697eaca768 parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c555a57f-0090-4a43-8091-a4333390327e_isrForCrosstalkSources_1031718_28.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c555a57f-0090-4a43-8091-a4333390327e_isrForCrosstalkSources_1031718_28.stderr parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 07635ac62bcdf9bda04f025c538400e8 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/18d55a88-b62d-4329-b315-6d15c17eab7c_isrForCrosstalkSources_1031705_36.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/18d55a88-b62d-4329-b315-6d15c17eab7c_isrForCrosstalkSources_1031705_36.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash c9e74d63651d0ca02a69e512c9be255d parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/14891035-47c8-4868-a7d9-5deb83468188_isrForCrosstalkSources_1031727_51.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/14891035-47c8-4868-a7d9-5deb83468188_isrForCrosstalkSources_1031727_51.stderr parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash b968fab45018ffa1fe1123c06d0a2c9c parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3d5ccbf6-f180-490a-9848-6ef1b8ca5c74_isrForCrosstalkSources_1031736_36.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3d5ccbf6-f180-490a-9848-6ef1b8ca5c74_isrForCrosstalkSources_1031736_36.stderr parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash e7b1e09ebfd0e12aa0215c4fba3bfe0e parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f54ba8a4-6b88-4925-938f-7a4b35fbfb70_isrForCrosstalkSources_1031709_27.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f54ba8a4-6b88-4925-938f-7a4b35fbfb70_isrForCrosstalkSources_1031709_27.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash ad328c84e889270d48a6d09475e1a5bf parsl.dataflow.memoization INFO: Task 391 had no result in cache 2024-03-17 04:02:01 proc_lsst.multi:170 [INFO] [multi] job_id 17147935 proc_lsst.multi INFO: [multi] job_id 17147935 2024-03-17 04:02:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:02:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147935 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147935 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17147935 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-17 04:02:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-17 04:02:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:02:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/b2bf4835-6b81-416c-908e-035772cbd953_isrForCrosstalkSources_1031735_9.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/b2bf4835-6b81-416c-908e-035772cbd953_isrForCrosstalkSources_1031735_9.stderr parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 0702ae8f3d2e98d3dfc929858fb70739 parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ebb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a61509b5-b0b5-49d7-a20b-620e2d867601_isrForCrosstalkSources_1031723_14.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a61509b5-b0b5-49d7-a20b-620e2d867601_isrForCrosstalkSources_1031723_14.stderr parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 113f03e82e8fa709f2b63834e2fd7d48 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8e9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/66883233-1f3b-4386-8a57-36265c0fa89b_isrForCrosstalkSources_1031717_11.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/66883233-1f3b-4386-8a57-36265c0fa89b_isrForCrosstalkSources_1031717_11.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 7c95d3fc5739af7affb3cc4289b33365 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/29810860-129b-4371-a17b-4124b5182956_isrForCrosstalkSources_1031703_42.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/29810860-129b-4371-a17b-4124b5182956_isrForCrosstalkSources_1031703_42.stderr parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash d1ae87b3224a3660e575e2756b92defa parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/397b9f52-f7ed-43fd-b71c-9d808b2f64f2_isrForCrosstalkSources_1031732_2.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/397b9f52-f7ed-43fd-b71c-9d808b2f64f2_isrForCrosstalkSources_1031732_2.stderr parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 1a2f98f92acb0bbb5641c8fc38c81275 parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/503faeed-9f51-4764-b1b8-a90d561fa9b6_isrForCrosstalkSources_1031707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/503faeed-9f51-4764-b1b8-a90d561fa9b6_isrForCrosstalkSources_1031707_61.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash a29799f4e67e64be73abdef9aa9ded72 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/12391a4e-d888-45bf-ad3e-bad7a7fd96ef_isrForCrosstalkSources_1031726_28.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/12391a4e-d888-45bf-ad3e-bad7a7fd96ef_isrForCrosstalkSources_1031726_28.stderr parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 3b15fa446f7ba0527a54466ac9e85cbe parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1d91ec54-4c6d-475c-875f-69f976334a03_isrForCrosstalkSources_1031733_2.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1d91ec54-4c6d-475c-875f-69f976334a03_isrForCrosstalkSources_1031733_2.stderr parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash c1eec5bff88feb17c2bbbdeaceb79a5c parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/042558c1-6b67-47d8-9187-1d400741c634_isrForCrosstalkSources_1031737_17.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/042558c1-6b67-47d8-9187-1d400741c634_isrForCrosstalkSources_1031737_17.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 50525d813ad4cf680a0600d67c0afbab parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/1dece97a-f403-4479-9698-8c8bf1316125_isrForCrosstalkSources_1031737_16.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/1dece97a-f403-4479-9698-8c8bf1316125_isrForCrosstalkSources_1031737_16.stderr parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 50fc7cadb74518deb97f8c1629246f6f parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3128c055-1904-4984-b122-1ca6eacdd8cc_isrForCrosstalkSources_1031729_5.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3128c055-1904-4984-b122-1ca6eacdd8cc_isrForCrosstalkSources_1031729_5.stderr parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 1a0fd4798e7473b4880b957dcc92b14b parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e6949071-de22-48ca-880e-cfb67d9f9e70_isrForCrosstalkSources_1031717_7.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e6949071-de22-48ca-880e-cfb67d9f9e70_isrForCrosstalkSources_1031717_7.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash e952a39f7c9cf244c1cb0c55e9a7bdbc parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7eb9181a-2aca-4735-8242-4e10d4976b34_isrForCrosstalkSources_1031714_23.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7eb9181a-2aca-4735-8242-4e10d4976b34_isrForCrosstalkSources_1031714_23.stderr parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 2a58d531f6267c52355ceda9489a8275 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/f386a45a-d188-4f0a-b3bb-5753c3228c71_isrForCrosstalkSources_1031717_62.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/f386a45a-d188-4f0a-b3bb-5753c3228c71_isrForCrosstalkSources_1031717_62.stderr parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash bce542245e0b9527ebbc5304679b0c53 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/1b2a5402-b27d-48f6-af04-4a7cb45afd48_isrForCrosstalkSources_1031707_34.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/1b2a5402-b27d-48f6-af04-4a7cb45afd48_isrForCrosstalkSources_1031707_34.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash fd88d388137ccbf619ee599087cf17fe parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/837dbff7-a150-423b-a8a4-2315ac37f30b_isrForCrosstalkSources_1031731_31.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/837dbff7-a150-423b-a8a4-2315ac37f30b_isrForCrosstalkSources_1031731_31.stderr parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 51d74483e875158b6f5ab991abfaecb6 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/29d4639e-f90f-424d-a7ee-35e4767fd29c_isrForCrosstalkSources_1031708_45.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/29d4639e-f90f-424d-a7ee-35e4767fd29c_isrForCrosstalkSources_1031708_45.stderr parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash c9e089b97419f5f007ff9d7a827bc065 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/9c854c7e-ee4f-4778-9fb7-c5524dc22930_isrForCrosstalkSources_1031719_30.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/9c854c7e-ee4f-4778-9fb7-c5524dc22930_isrForCrosstalkSources_1031719_30.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash bdd38719e804b2c1c056dc70a05aad40 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/46f94bc4-360e-471c-8478-a28a30218c27_isrForCrosstalkSources_1031705_1.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/46f94bc4-360e-471c-8478-a28a30218c27_isrForCrosstalkSources_1031705_1.stderr parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 58b7e26934dc17cb58dedee8606c280b parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/ff9c20dd-582a-442e-b9e0-27a9358de369_isrForCrosstalkSources_1031706_8.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/ff9c20dd-582a-442e-b9e0-27a9358de369_isrForCrosstalkSources_1031706_8.stderr parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 8e8a597f333f25a8835af6069a49e7ba parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1c1820ac-8197-40d0-a6b1-8e07d8ac3e92_isrForCrosstalkSources_1031732_45.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1c1820ac-8197-40d0-a6b1-8e07d8ac3e92_isrForCrosstalkSources_1031732_45.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 38e33c58f2d9b5c29ef6241885dc5577 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/974d8a0c-6291-40c1-a7c6-8585a16c5eed_isrForCrosstalkSources_1031712_6.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/974d8a0c-6291-40c1-a7c6-8585a16c5eed_isrForCrosstalkSources_1031712_6.stderr parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 488de7f1fcf9296124bcd89421d48713 parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8e5a172d-2e77-4922-94a1-715480689bea_isrForCrosstalkSources_1031704_1.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8e5a172d-2e77-4922-94a1-715480689bea_isrForCrosstalkSources_1031704_1.stderr parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 7665966965d6ed189c8c1c912c951122 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bb4656ed-de8f-4b48-8a7e-7143c244cab0_isrForCrosstalkSources_1031729_15.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bb4656ed-de8f-4b48-8a7e-7143c244cab0_isrForCrosstalkSources_1031729_15.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 596dbde70c87a474f12f6278d470c978 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/9bb5ca01-d13e-4bce-b856-5e04b8a6668b_isrForCrosstalkSources_1031710_10.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/9bb5ca01-d13e-4bce-b856-5e04b8a6668b_isrForCrosstalkSources_1031710_10.stderr parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash 537c7ca48d9f461fbb9b6d1491aab071 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/97666d4e-d27c-44a5-87be-47475d887fd7_isrForCrosstalkSources_1031730_59.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/97666d4e-d27c-44a5-87be-47475d887fd7_isrForCrosstalkSources_1031730_59.stderr parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 31bd4111c0f8b0004b01b2b52bb502c9 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/86b33649-ad5d-4c9e-a78d-139c0517c014_isrForCrosstalkSources_1031721_24.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/86b33649-ad5d-4c9e-a78d-139c0517c014_isrForCrosstalkSources_1031721_24.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 026713aef59909fa38348ac94913099f parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a328aeb7-4554-4c70-bb11-4960509a0189_isrForCrosstalkSources_1031726_33.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a328aeb7-4554-4c70-bb11-4960509a0189_isrForCrosstalkSources_1031726_33.stderr parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 7026d02efd217ba240a1ca47d3e052c0 parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/912c2a58-f1f7-45e4-9e98-ebc6968c35f0_isrForCrosstalkSources_1031711_33.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/912c2a58-f1f7-45e4-9e98-ebc6968c35f0_isrForCrosstalkSources_1031711_33.stderr parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash eefadfbc783642f4b58f321170a48001 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/cc1f9671-19ec-45fc-9e36-aca93bae5529_isrForCrosstalkSources_1031737_5.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/cc1f9671-19ec-45fc-9e36-aca93bae5529_isrForCrosstalkSources_1031737_5.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 2bfd5ec19941ae9ef5867df0dfce0c44 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cb45e38d-a9cc-40b6-8848-5a6ca8bd7f5c_isrForCrosstalkSources_1031707_9.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cb45e38d-a9cc-40b6-8848-5a6ca8bd7f5c_isrForCrosstalkSources_1031707_9.stderr parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash f29705e003aff35d3f0fa29cd33b6d36 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/a9cc6abb-01e8-464d-81cd-e6a2a532d2bc_isrForCrosstalkSources_1031732_27.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/a9cc6abb-01e8-464d-81cd-e6a2a532d2bc_isrForCrosstalkSources_1031732_27.stderr parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 048175279618206f8c1c319a4375db4f parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/84750fd8-82c4-4a76-adef-9dc9a64ef17c_isrForCrosstalkSources_1031736_40.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/84750fd8-82c4-4a76-adef-9dc9a64ef17c_isrForCrosstalkSources_1031736_40.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 4ed0bf8bce21621538466e68fa8d2b5e parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7728ed00-1ec8-45f6-8430-b43ae6165584_isrForCrosstalkSources_1031721_60.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7728ed00-1ec8-45f6-8430-b43ae6165584_isrForCrosstalkSources_1031721_60.stderr parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash d8e35855e50fd00e62ef4cd78c88fb03 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d7aa2d4b-9238-44e5-99f5-019764231df5_isrForCrosstalkSources_1031703_46.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d7aa2d4b-9238-44e5-99f5-019764231df5_isrForCrosstalkSources_1031703_46.stderr parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 64ec2059c3719a91b6b5df77cd8f152e parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/59261fd3-1b45-48d3-af3e-dda86af19ea9_isrForCrosstalkSources_1031714_38.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/59261fd3-1b45-48d3-af3e-dda86af19ea9_isrForCrosstalkSources_1031714_38.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 7500828a9c50b4e86ee64d5db0c70bc8 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/40f6f468-39bc-4984-9fc8-770089a0f929_isrForCrosstalkSources_1031713_7.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/40f6f468-39bc-4984-9fc8-770089a0f929_isrForCrosstalkSources_1031713_7.stderr parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 36444d9bc4a0c307c2f81cab89a608ea parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/5ef3d934-4ae5-4199-a4da-c8850da5f87b_isrForCrosstalkSources_1031699_53.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/5ef3d934-4ae5-4199-a4da-c8850da5f87b_isrForCrosstalkSources_1031699_53.stderr parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash adbcfd57b4645aaeb256155e89e21bb6 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/eb9c002b-cb02-4919-864a-c118dc889c5a_isrForCrosstalkSources_1031718_33.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/eb9c002b-cb02-4919-864a-c118dc889c5a_isrForCrosstalkSources_1031718_33.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 9587c6b9e7abf8f9074a65a6631c8fe0 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8f7267c9-970d-49ab-9850-8059da3fb7e4_isrForCrosstalkSources_1031704_56.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8f7267c9-970d-49ab-9850-8059da3fb7e4_isrForCrosstalkSources_1031704_56.stderr parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash ba8fe74bf81d20300fceec55d4bb93a7 parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f5304999-c6f1-4b4a-87f5-d3ba74a60ad4_isrForCrosstalkSources_1031705_7.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f5304999-c6f1-4b4a-87f5-d3ba74a60ad4_isrForCrosstalkSources_1031705_7.stderr parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 1ff12bb7c5b6544bee652028ca788ebf parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/b6510bb2-f96d-4b68-b136-626c7abe8a00_isrForCrosstalkSources_1031737_62.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/b6510bb2-f96d-4b68-b136-626c7abe8a00_isrForCrosstalkSources_1031737_62.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 8f6a830ed31bb75441f41d5fa40f8506 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/70207f8c-1d52-48b0-a646-d58d0f695fd6_isrForCrosstalkSources_1031734_13.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/70207f8c-1d52-48b0-a646-d58d0f695fd6_isrForCrosstalkSources_1031734_13.stderr parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash cad25c634e052543f98443958fbdcf70 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/de471be6-f7ea-423b-ae8e-d61e4fd67370_isrForCrosstalkSources_1031711_15.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/de471be6-f7ea-423b-ae8e-d61e4fd67370_isrForCrosstalkSources_1031711_15.stderr parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 6bff0eb002317bbe8475917db767a6a5 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/95ef499f-9dcc-4516-93ea-e82539f55cab_isrForCrosstalkSources_1031708_42.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/95ef499f-9dcc-4516-93ea-e82539f55cab_isrForCrosstalkSources_1031708_42.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash d329e29ed92b77882839a44bae15640b parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e0a1e611-3397-46f2-ac25-0b11fffe6dbb_isrForCrosstalkSources_1031728_2.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e0a1e611-3397-46f2-ac25-0b11fffe6dbb_isrForCrosstalkSources_1031728_2.stderr parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash aabbe740ab73f641825c4f8ff5afeb3f parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/daeb1dc8-f377-4b3d-8b59-591aaa012b67_isrForCrosstalkSources_1031731_1.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/daeb1dc8-f377-4b3d-8b59-591aaa012b67_isrForCrosstalkSources_1031731_1.stderr parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash 8cc8ababfb47a10fdb25ead137d0614e parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3d16014f-8805-418a-81b5-61f6d9c6b92d_isrForCrosstalkSources_1031732_16.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3d16014f-8805-418a-81b5-61f6d9c6b92d_isrForCrosstalkSources_1031732_16.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash dfb1a8df04fa9784bfb5706a3249d6e5 parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/905a0aba-4066-4147-8027-260d73565b30_isrForCrosstalkSources_1031730_51.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/905a0aba-4066-4147-8027-260d73565b30_isrForCrosstalkSources_1031730_51.stderr parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash a17f022ca6134cc70cfd0e7d8a45706c parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/f994ced1-20f3-40d5-8140-76dc0dc29ea8_isrForCrosstalkSources_1031738_4.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/f994ced1-20f3-40d5-8140-76dc0dc29ea8_isrForCrosstalkSources_1031738_4.stderr parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 5686fd07a1ecea0eb5b0aea4c62feea0 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/1e94e6d4-dfaa-498a-9d7d-4b0dd027593b_isrForCrosstalkSources_1031731_22.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/1e94e6d4-dfaa-498a-9d7d-4b0dd027593b_isrForCrosstalkSources_1031731_22.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 1fd258548b597fd90bca2d380f241704 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/15cb866a-b6cd-45a6-8213-d6009c15d862_isrForCrosstalkSources_1031727_62.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/15cb866a-b6cd-45a6-8213-d6009c15d862_isrForCrosstalkSources_1031727_62.stderr parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash baa2cf532f4eee6d99cfa8e7c4014a8e parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/ba621ef7-791c-42d0-b3e5-2d3790036eeb_isrForCrosstalkSources_1031733_48.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/ba621ef7-791c-42d0-b3e5-2d3790036eeb_isrForCrosstalkSources_1031733_48.stderr parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash ffba587f6bcc1525a909938f8a8beb4a parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4111cc59-bd18-4588-8728-d43e4b1b794d_isrForCrosstalkSources_1031699_17.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4111cc59-bd18-4588-8728-d43e4b1b794d_isrForCrosstalkSources_1031699_17.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 941d60ce9a1698c404db9be1f9d8a45c parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/8762c009-2bee-4aaa-81cb-4ee17575302e_isrForCrosstalkSources_1031738_3.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/8762c009-2bee-4aaa-81cb-4ee17575302e_isrForCrosstalkSources_1031738_3.stderr parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash 187917ac28bf810c4f94cbd2bda2a291 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ea37b81f-d10a-43b9-ab4d-5c71eee7479d_isrForCrosstalkSources_1031737_23.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ea37b81f-d10a-43b9-ab4d-5c71eee7479d_isrForCrosstalkSources_1031737_23.stderr parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 94817f11475b5b9be8176b358410d252 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/9d1ba715-34c9-477f-9b14-7ee8e05a0986_isrForCrosstalkSources_1031701_58.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/9d1ba715-34c9-477f-9b14-7ee8e05a0986_isrForCrosstalkSources_1031701_58.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash dbc11ff3ef9e29ba46025e232fbc3fed parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/9d23937a-c75e-4a98-b72a-5f0abd2530c4_isrForCrosstalkSources_1031714_52.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/9d23937a-c75e-4a98-b72a-5f0abd2530c4_isrForCrosstalkSources_1031714_52.stderr parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash bc21afeaff4dc19a60af74c0b22737b1 parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1765bdc2-1794-4cc7-9694-796eeaa7ff74_isrForCrosstalkSources_1031715_46.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1765bdc2-1794-4cc7-9694-796eeaa7ff74_isrForCrosstalkSources_1031715_46.stderr parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 5abc84218bfe2e162ea8f0aaa5c7ae95 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/112cf319-cfc5-4d7f-8fcf-1c96864999ba_isrForCrosstalkSources_1031733_38.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/112cf319-cfc5-4d7f-8fcf-1c96864999ba_isrForCrosstalkSources_1031733_38.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 213a3e8ea1f47a4a4565da240b9ea08f parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ad2d41c9-ff7c-4a01-b76a-b1f9e7715582_isrForCrosstalkSources_1031725_37.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ad2d41c9-ff7c-4a01-b76a-b1f9e7715582_isrForCrosstalkSources_1031725_37.stderr parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 50662fb283fad1425b8b5809468427e7 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/94e9261b-abf2-413f-b6c9-eb473f619b98_isrForCrosstalkSources_1031717_23.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/94e9261b-abf2-413f-b6c9-eb473f619b98_isrForCrosstalkSources_1031717_23.stderr parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash bef29c27803863dabb707373c9d06c4d parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/4d3caa72-92b8-4590-bbf1-76dae8401922_isrForCrosstalkSources_1031705_21.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/4d3caa72-92b8-4590-bbf1-76dae8401922_isrForCrosstalkSources_1031705_21.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 4d0473079c3c780344199527e89c3529 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/dd90c357-9ac7-450b-917a-e4e09ca97d2f_isrForCrosstalkSources_1031708_15.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/dd90c357-9ac7-450b-917a-e4e09ca97d2f_isrForCrosstalkSources_1031708_15.stderr parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 589a0447413e1d11d802a0c30b2ebe12 parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/08b4e6db-1894-493f-88a2-9a16ef927bc5_isrForCrosstalkSources_1031730_6.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/08b4e6db-1894-493f-88a2-9a16ef927bc5_isrForCrosstalkSources_1031730_6.stderr parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 1c8f2338956fe6940fcccef65fb2fb40 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67a3aa5e-51e9-4b4e-ac0c-9d0fec4bcb38_isrForCrosstalkSources_1031728_57.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67a3aa5e-51e9-4b4e-ac0c-9d0fec4bcb38_isrForCrosstalkSources_1031728_57.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 7d2fbd9fa11bdc5ae035452faa96585d parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e90ae8d1-c8b1-4679-a1ad-d28d057f91c2_isrForCrosstalkSources_1031712_29.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e90ae8d1-c8b1-4679-a1ad-d28d057f91c2_isrForCrosstalkSources_1031712_29.stderr parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 264b66f461a67734b3596b34194b064e parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/3e915396-68d7-417e-99a5-c9269afc39cd_isrForCrosstalkSources_1031724_46.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/3e915396-68d7-417e-99a5-c9269afc39cd_isrForCrosstalkSources_1031724_46.stderr parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash 60fa3b631b2e8bb0b9b14a6bf8687529 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/f8ea7daa-1e7f-4e7a-90b6-5462429691c1_isrForCrosstalkSources_1031725_22.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/f8ea7daa-1e7f-4e7a-90b6-5462429691c1_isrForCrosstalkSources_1031725_22.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 40693cab575329036ea5424fa88c44a2 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/79ab02c6-d619-4eea-a5e1-9097a83e6e60_isrForCrosstalkSources_1031734_2.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/79ab02c6-d619-4eea-a5e1-9097a83e6e60_isrForCrosstalkSources_1031734_2.stderr parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 0520a2cccecf209beb2b3a14d9b4961a parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c10f3430-99c6-46c0-86b6-af6e9f7eafa4_isrForCrosstalkSources_1031734_39.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c10f3430-99c6-46c0-86b6-af6e9f7eafa4_isrForCrosstalkSources_1031734_39.stderr parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 4267f3b71b946651f4c1452e7edea181 parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7bcbfe50-e411-459e-bb9f-0fd52acf78c4_isrForCrosstalkSources_1031735_10.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7bcbfe50-e411-459e-bb9f-0fd52acf78c4_isrForCrosstalkSources_1031735_10.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash aa191b3b8b944f9638c87186848d43a9 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/d8e99695-3bc8-4ce2-9c2c-c9486dc25e8d_isrForCrosstalkSources_1031705_35.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/d8e99695-3bc8-4ce2-9c2c-c9486dc25e8d_isrForCrosstalkSources_1031705_35.stderr parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 459b4ee6d896e97e297314ef487ec6a5 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8b2d1ecb-4b61-46d4-b296-57bd12186f16_isrForCrosstalkSources_1031733_59.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8b2d1ecb-4b61-46d4-b296-57bd12186f16_isrForCrosstalkSources_1031733_59.stderr parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash f21d993b5b9f27bd541fc3601babb74b parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/252ee987-19bd-4cd4-b1d0-3dccb238df32_isrForCrosstalkSources_1031733_55.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/252ee987-19bd-4cd4-b1d0-3dccb238df32_isrForCrosstalkSources_1031733_55.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 77b7d8f7a16d12f50564662bd052090b parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/9963c37c-5a3b-4dd9-9070-b3e83dbab6ee_isrForCrosstalkSources_1031736_15.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/9963c37c-5a3b-4dd9-9070-b3e83dbab6ee_isrForCrosstalkSources_1031736_15.stderr parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash deeb876a6fc146925276f3d514c56bb0 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:02 proc_lsst.multi:170 [INFO] [multi] job_id 17147936 proc_lsst.multi INFO: [multi] job_id 17147936 2024-03-17 04:02:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 04:02:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147936 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147936 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17147936 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-17 04:02:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-17 04:02:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 04:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0722407d-bd59-46ec-84eb-15598f018e7d_isrForCrosstalkSources_1031714_14.stdout parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0722407d-bd59-46ec-84eb-15598f018e7d_isrForCrosstalkSources_1031714_14.stderr parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash e64ea1a119b48c22251ac0e86b6eb9f5 parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/af12f0e1-ba0a-4726-8c64-620356812c82_isrForCrosstalkSources_1031737_37.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/af12f0e1-ba0a-4726-8c64-620356812c82_isrForCrosstalkSources_1031737_37.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 027e8ff65dafe1d51f73be119eeefb0f parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/5e09c646-f250-47ac-92cd-357e49748867_isrForCrosstalkSources_1031708_8.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/5e09c646-f250-47ac-92cd-357e49748867_isrForCrosstalkSources_1031708_8.stderr parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 03a07187b59bc0d5deaccadf25648305 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ccccfe57-0341-4b68-a436-bd4a1442ca68_isrForCrosstalkSources_1031726_15.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ccccfe57-0341-4b68-a436-bd4a1442ca68_isrForCrosstalkSources_1031726_15.stderr parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 9d0ba444df4a6072f25ce888104d24f8 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/90eb9ac1-853d-4a9a-9193-6a66201c50f0_isrForCrosstalkSources_1031706_9.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/90eb9ac1-853d-4a9a-9193-6a66201c50f0_isrForCrosstalkSources_1031706_9.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 887a264816949b0ea682fc4f4275e008 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/b1205a50-85b2-4669-bb8d-dc255b577277_isrForCrosstalkSources_1031734_37.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/b1205a50-85b2-4669-bb8d-dc255b577277_isrForCrosstalkSources_1031734_37.stderr parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 6b2fce76d94c7f73860dbd5f7ec76d7a parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/de2881a7-bd1d-4b7d-957a-774fb4c049d6_isrForCrosstalkSources_1031704_12.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/de2881a7-bd1d-4b7d-957a-774fb4c049d6_isrForCrosstalkSources_1031704_12.stderr parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash 95b22a6f18c52ac23b2c4c204be4db38 parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c0d6bcec-eea3-403d-a429-e9e3c83fc3dd_isrForCrosstalkSources_1031703_3.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c0d6bcec-eea3-403d-a429-e9e3c83fc3dd_isrForCrosstalkSources_1031703_3.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 96d0bf70f7edb85e8baf5e7c86186873 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6e95fc96-2cba-4660-9045-08905af852dc_isrForCrosstalkSources_1031728_38.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6e95fc96-2cba-4660-9045-08905af852dc_isrForCrosstalkSources_1031728_38.stderr parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash e4129652d6d9bda6b28d0039d4b455bb parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/5cfacb1e-c223-4046-8fb1-8375d6eb63b6_isrForCrosstalkSources_1031738_50.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/5cfacb1e-c223-4046-8fb1-8375d6eb63b6_isrForCrosstalkSources_1031738_50.stderr parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash b235b73c75a03fc590f19febe4659f23 parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/926eb105-5f4c-45f2-a511-a97e836ad649_isrForCrosstalkSources_1031699_36.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/926eb105-5f4c-45f2-a511-a97e836ad649_isrForCrosstalkSources_1031699_36.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash bb912293cb9204d005027cf14af4288d parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c0278e48-e9ce-4945-8599-107c91373ba7_isrForCrosstalkSources_1031710_20.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c0278e48-e9ce-4945-8599-107c91373ba7_isrForCrosstalkSources_1031710_20.stderr parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 0fd15dfd73093399f5ee93897bcf4b33 parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cfe8e82b-dd7b-4a3e-9f13-d72d694513da_isrForCrosstalkSources_1031716_35.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cfe8e82b-dd7b-4a3e-9f13-d72d694513da_isrForCrosstalkSources_1031716_35.stderr parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 6a96f904843978866457f6a62c898c5e parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8d0806e8-ed61-4c31-8151-6f18ad51e45b_isrForCrosstalkSources_1031730_10.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8d0806e8-ed61-4c31-8151-6f18ad51e45b_isrForCrosstalkSources_1031730_10.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 2ca72083ea9bac047b49680b5001f1e9 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/cbd00b3b-93ec-486c-bc23-050819a4e274_isrForCrosstalkSources_1031731_36.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/cbd00b3b-93ec-486c-bc23-050819a4e274_isrForCrosstalkSources_1031731_36.stderr parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash bed328bf106b4ad50803ae0f18d8def8 parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a2abeaba-8387-403d-88b9-b557983543fa_isrForCrosstalkSources_1031730_49.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a2abeaba-8387-403d-88b9-b557983543fa_isrForCrosstalkSources_1031730_49.stderr parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash e1e0d5419793af7b3ee3286a7c5a1879 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/a6258bc9-cbdf-40f4-916b-d397111c3e41_isrForCrosstalkSources_1031733_44.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/a6258bc9-cbdf-40f4-916b-d397111c3e41_isrForCrosstalkSources_1031733_44.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash 9a44544ce45b7610a6eff34ce9f7f456 parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/077787f9-c7b7-4b55-bf99-75eddb59cb66_isrForCrosstalkSources_1031715_2.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/077787f9-c7b7-4b55-bf99-75eddb59cb66_isrForCrosstalkSources_1031715_2.stderr parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 9f21b0ad8909110d5b5a03b6879bf2b6 parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/f4202191-185d-42d8-b034-27a5376eef15_isrForCrosstalkSources_1031707_17.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/f4202191-185d-42d8-b034-27a5376eef15_isrForCrosstalkSources_1031707_17.stderr parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash a7e66a07dbbd07f5c4101b58a0d88dcf parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0e644181-e901-4baa-ae58-15d5e91da8c2_isrForCrosstalkSources_1031724_22.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0e644181-e901-4baa-ae58-15d5e91da8c2_isrForCrosstalkSources_1031724_22.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 39fe4481268a0be7db73c823b69d7623 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8a00e4bb-f258-4f3b-96ee-7703fd088c1e_isrForCrosstalkSources_1031704_47.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8a00e4bb-f258-4f3b-96ee-7703fd088c1e_isrForCrosstalkSources_1031704_47.stderr parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash dd6240da7504ccfe3c0e5c349a3aa59f parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0041d2f2-8fcb-40d7-9146-2a39346a1f61_isrForCrosstalkSources_1031719_20.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0041d2f2-8fcb-40d7-9146-2a39346a1f61_isrForCrosstalkSources_1031719_20.stderr parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 03fbb2b31171a8be06f97d6f76ecccdf parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/acbaf117-9f31-41b3-923d-4f438efed85b_isrForCrosstalkSources_1031731_33.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/acbaf117-9f31-41b3-923d-4f438efed85b_isrForCrosstalkSources_1031731_33.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 62fbeaaf32e52949a846349dcc1d8f6e parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6249170e-b387-4b79-aa5a-88e614450a40_isrForCrosstalkSources_1031728_5.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6249170e-b387-4b79-aa5a-88e614450a40_isrForCrosstalkSources_1031728_5.stderr parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 39b444742c108b3b3f34d190409af5c3 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/58dd09dd-08eb-491f-ae36-98912e475d12_isrForCrosstalkSources_1031709_34.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/58dd09dd-08eb-491f-ae36-98912e475d12_isrForCrosstalkSources_1031709_34.stderr parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 3ce612c1172c617ba9660195abc4c6c7 parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c23e5d1b-af57-4a38-94b1-cb655c1a5432_isrForCrosstalkSources_1031726_55.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c23e5d1b-af57-4a38-94b1-cb655c1a5432_isrForCrosstalkSources_1031726_55.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 0ca3dcce5183640f7dc5e7a3bd7f34e5 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3aa42ae6-5be0-44f7-bb3a-1748785b1e5f_isrForCrosstalkSources_1031735_3.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3aa42ae6-5be0-44f7-bb3a-1748785b1e5f_isrForCrosstalkSources_1031735_3.stderr parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 72990ec6d1fbbc9c9b7620078386c867 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/aac93c07-3601-44f8-93e3-97abd5511e3d_isrForCrosstalkSources_1031702_6.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/aac93c07-3601-44f8-93e3-97abd5511e3d_isrForCrosstalkSources_1031702_6.stderr parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash d1bd76966bd6f94c19dc268c827bb51e parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/6a4c2764-4cbb-45d1-be13-7b5492401785_isrForCrosstalkSources_1031736_5.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/6a4c2764-4cbb-45d1-be13-7b5492401785_isrForCrosstalkSources_1031736_5.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 434fa0581b231fcc771704f175fd3f52 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a477c1f4-6fae-4876-a3e2-fe3717e74258_isrForCrosstalkSources_1031719_55.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a477c1f4-6fae-4876-a3e2-fe3717e74258_isrForCrosstalkSources_1031719_55.stderr parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 7cfc9353121c7ee6356cf5dfda14c175 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/33b91735-e8bb-4ab4-b4ca-d2188f3d1089_isrForCrosstalkSources_1031729_19.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/33b91735-e8bb-4ab4-b4ca-d2188f3d1089_isrForCrosstalkSources_1031729_19.stderr parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 1942d2a209d0bc7a358194d213877659 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/50c6b3c4-5084-4469-aaa6-2984846742db_isrForCrosstalkSources_1031700_3.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/50c6b3c4-5084-4469-aaa6-2984846742db_isrForCrosstalkSources_1031700_3.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 5f1abf8794ebaa8ff3c7813c7ad08775 parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/3cece70b-72be-4fae-9776-3185cc8988bb_isrForCrosstalkSources_1031703_5.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/3cece70b-72be-4fae-9776-3185cc8988bb_isrForCrosstalkSources_1031703_5.stderr parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 99babda8b7486bd8d9300a738309e90b parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/05d4e54b-dad1-4e6f-8ec8-2958bd6ff2c7_isrForCrosstalkSources_1031735_49.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/05d4e54b-dad1-4e6f-8ec8-2958bd6ff2c7_isrForCrosstalkSources_1031735_49.stderr parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash af5df31ffd2e4b593342031e86a782bd parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/22818813-1eed-43ba-ab9b-d72edfd5086c_isrForCrosstalkSources_1031716_1.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/22818813-1eed-43ba-ab9b-d72edfd5086c_isrForCrosstalkSources_1031716_1.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash fa9acf3dad3753f9e3bf7273ba8ca240 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/4dba6235-aa47-4413-a7d7-cdf76114c59a_isrForCrosstalkSources_1031708_17.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/4dba6235-aa47-4413-a7d7-cdf76114c59a_isrForCrosstalkSources_1031708_17.stderr parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 7e044abd2955a6f728b88f40030f5972 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/83704301-57b8-482d-9fe6-7e93804104bf_isrForCrosstalkSources_1031718_16.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/83704301-57b8-482d-9fe6-7e93804104bf_isrForCrosstalkSources_1031718_16.stderr parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash d0487d818b7ee6d923a67a9d2ba97b0b parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/48f03e26-bcb6-44db-b701-378aa93aff66_isrForCrosstalkSources_1031722_40.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/48f03e26-bcb6-44db-b701-378aa93aff66_isrForCrosstalkSources_1031722_40.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 96ad0b2a4b0bd249071f544c7e57d8fb parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5e4b08ad-9df0-43ce-98a2-70d4c938a7cb_isrForCrosstalkSources_1031736_1.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5e4b08ad-9df0-43ce-98a2-70d4c938a7cb_isrForCrosstalkSources_1031736_1.stderr parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 8d109fb8d8136b45a6dbf909b492cd90 parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/a6947acd-3746-4a62-bdff-01a3bf4d1319_isrForCrosstalkSources_1031727_10.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/a6947acd-3746-4a62-bdff-01a3bf4d1319_isrForCrosstalkSources_1031727_10.stderr parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash a0d109a66502f5737686060bfc730267 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e793c485-16b1-4afb-8385-297a67bfae4a_isrForCrosstalkSources_1031731_39.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e793c485-16b1-4afb-8385-297a67bfae4a_isrForCrosstalkSources_1031731_39.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash ac2f91196a11e858b197a4e812012367 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/9c03017d-cc9e-4c14-b456-8b9a4394c594_isrForCrosstalkSources_1031738_15.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/9c03017d-cc9e-4c14-b456-8b9a4394c594_isrForCrosstalkSources_1031738_15.stderr parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 8fd626fc50ac8cd09d62b44b93d2c279 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3218ab54-7ef5-478e-ace6-a4c67284e9e3_isrForCrosstalkSources_1031710_32.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3218ab54-7ef5-478e-ace6-a4c67284e9e3_isrForCrosstalkSources_1031710_32.stderr parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 7fa8dc780370f822fa0b8d39e407ad0e parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/85a9bdf3-2db2-4e16-9b0a-63b82470d58f_isrForCrosstalkSources_1031735_15.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/85a9bdf3-2db2-4e16-9b0a-63b82470d58f_isrForCrosstalkSources_1031735_15.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 226beb9181280e83f68ae4cfaff1c7d7 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cb45c076-8aad-4a6e-b6c4-794733e2d2c0_isrForCrosstalkSources_1031707_42.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cb45c076-8aad-4a6e-b6c4-794733e2d2c0_isrForCrosstalkSources_1031707_42.stderr parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 0ea3fd46edf7ef3b2753759f48bdd729 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/f8903d6f-2d29-4037-86fa-98fec552c472_isrForCrosstalkSources_1031704_8.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/f8903d6f-2d29-4037-86fa-98fec552c472_isrForCrosstalkSources_1031704_8.stderr parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash b4790e34fed198a0ca0ef24de0831624 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/db71bdd1-05f3-40fa-9bab-3767b2015e63_isrForCrosstalkSources_1031709_8.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/db71bdd1-05f3-40fa-9bab-3767b2015e63_isrForCrosstalkSources_1031709_8.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 5c82beb2b872e28dc41204046a8e8d00 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e5a3309b-7de1-4297-b95e-78d02e89d177_isrForCrosstalkSources_1031709_58.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e5a3309b-7de1-4297-b95e-78d02e89d177_isrForCrosstalkSources_1031709_58.stderr parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash 088be356b187bec0314d84e66c00d641 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5fe09a30-8604-488a-b3e6-2ae88ea2e213_isrForCrosstalkSources_1031720_61.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5fe09a30-8604-488a-b3e6-2ae88ea2e213_isrForCrosstalkSources_1031720_61.stderr parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 572f1e9ac81fac6b47d0ad1999f540a1 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1cee8586-8e6a-4661-bdff-33a3193f136c_isrForCrosstalkSources_1031715_36.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1cee8586-8e6a-4661-bdff-33a3193f136c_isrForCrosstalkSources_1031715_36.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash 97569b1ab6628ca971b3f1e40420455a parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/2c1ab183-ab9c-459f-b3b9-f6d5f441f97a_isrForCrosstalkSources_1031730_25.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/2c1ab183-ab9c-459f-b3b9-f6d5f441f97a_isrForCrosstalkSources_1031730_25.stderr parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 91f6f4f5c7b172cde9917d0bf2f307a9 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:03 proc_lsst.multi:170 [INFO] [multi] job_id 17147937 proc_lsst.multi INFO: [multi] job_id 17147937 2024-03-17 04:02:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 04:02:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147937 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147937 parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 519 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17147937 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7706f961-e9a5-4779-8764-b4a8eaf76a99_isrForCrosstalkSources_1031721_25.stdout parsl.executors.status_handling INFO: Allocated block ID 8 parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7706f961-e9a5-4779-8764-b4a8eaf76a99_isrForCrosstalkSources_1031721_25.stderr parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-17 04:02:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-17 04:02:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 04:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash a50bf8b43f148eac8dea16713c8c6654 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e107100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2ddd0b38-1851-4980-ac9c-48bca0f37a79_isrForCrosstalkSources_1031726_14.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2ddd0b38-1851-4980-ac9c-48bca0f37a79_isrForCrosstalkSources_1031726_14.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash d9d7933d4b0fd76489c28d004ad0b6dd parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f23c306c-5ecc-4a66-8027-782a16d9931b_isrForCrosstalkSources_1031737_39.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f23c306c-5ecc-4a66-8027-782a16d9931b_isrForCrosstalkSources_1031737_39.stderr parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 5074a8763f173251b42a46797a7666e4 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/345b8153-8b1a-4f08-85bb-34b35a1c0e3d_isrForCrosstalkSources_1031704_38.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/345b8153-8b1a-4f08-85bb-34b35a1c0e3d_isrForCrosstalkSources_1031704_38.stderr parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 0a1113c0422c7fab50fde3a9ea797264 parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d499498c-d0e6-4ccb-9464-9399bc011fff_isrForCrosstalkSources_1031722_17.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d499498c-d0e6-4ccb-9464-9399bc011fff_isrForCrosstalkSources_1031722_17.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 6c00baefb3dfa000137951fdca088065 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/db7f0be4-57da-4612-89ab-3c1bff6e95c6_isrForCrosstalkSources_1031725_56.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/db7f0be4-57da-4612-89ab-3c1bff6e95c6_isrForCrosstalkSources_1031725_56.stderr parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 010884bb5471835704f94c3e74ea4311 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2fa0b349-9ef8-4a68-921b-0cd87c3db342_isrForCrosstalkSources_1031727_32.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2fa0b349-9ef8-4a68-921b-0cd87c3db342_isrForCrosstalkSources_1031727_32.stderr parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 845318d88c1eb3522595df500626ee19 parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d53faf69-6c41-4a43-8d15-4314e8a75442_isrForCrosstalkSources_1031727_44.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d53faf69-6c41-4a43-8d15-4314e8a75442_isrForCrosstalkSources_1031727_44.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash a6e25b9fe8a676369521bd3176f7c1d1 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/53e29225-e91b-4969-85f3-cb107f175b31_isrForCrosstalkSources_1031734_41.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/53e29225-e91b-4969-85f3-cb107f175b31_isrForCrosstalkSources_1031734_41.stderr parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 19a1f9875c339f2405342186a66871c4 parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/c177d6f0-2640-43cd-af41-4ac4ad5767f8_isrForCrosstalkSources_1031735_42.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/c177d6f0-2640-43cd-af41-4ac4ad5767f8_isrForCrosstalkSources_1031735_42.stderr parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash ac81b9113a9388442cb9a52b8aaefc2f parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5793b370-46bf-423d-9b84-958fa5db5497_isrForCrosstalkSources_1031705_14.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5793b370-46bf-423d-9b84-958fa5db5497_isrForCrosstalkSources_1031705_14.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash db22765507179c2dafdd6ff93ef814bb parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/7871b96c-e5d8-4111-b88a-af53b1e3fdf0_isrForCrosstalkSources_1031711_39.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/7871b96c-e5d8-4111-b88a-af53b1e3fdf0_isrForCrosstalkSources_1031711_39.stderr parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 030019fd361b89a211813ba2f4dd475c parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/97f68eee-37a1-4e7b-b376-2287a714d29a_isrForCrosstalkSources_1031711_4.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/97f68eee-37a1-4e7b-b376-2287a714d29a_isrForCrosstalkSources_1031711_4.stderr parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash b46c0736b2bb685a4296c7da40537b72 parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a5ea8139-636c-41f4-833d-5537f13c16d1_isrForCrosstalkSources_1031738_14.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a5ea8139-636c-41f4-833d-5537f13c16d1_isrForCrosstalkSources_1031738_14.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash aab7db67e82658e6c33472913729f616 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/1227ccb6-1fb7-43a8-9cc2-5c3fe779c957_isrForCrosstalkSources_1031714_48.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/1227ccb6-1fb7-43a8-9cc2-5c3fe779c957_isrForCrosstalkSources_1031714_48.stderr parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 1652286a86a28f31c7ccf31298e4e238 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a030ee6d-c073-423e-9b42-69c1ec14b48c_isrForCrosstalkSources_1031723_38.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a030ee6d-c073-423e-9b42-69c1ec14b48c_isrForCrosstalkSources_1031723_38.stderr parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash df1741fad637363eff777595520cd33f parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/d518331a-5af2-4267-b959-931f80b4d335_isrForCrosstalkSources_1031700_15.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/d518331a-5af2-4267-b959-931f80b4d335_isrForCrosstalkSources_1031700_15.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash c7b072377ab60df156c43522d62cf20b parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e3e611a5-c23c-4d7c-a898-16d6a16dba8c_isrForCrosstalkSources_1031717_1.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e3e611a5-c23c-4d7c-a898-16d6a16dba8c_isrForCrosstalkSources_1031717_1.stderr parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 47317961632a3fc01d4630b2103edfaa parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0760b9cd-0ebc-4823-9930-b0febb6e658d_isrForCrosstalkSources_1031720_62.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0760b9cd-0ebc-4823-9930-b0febb6e658d_isrForCrosstalkSources_1031720_62.stderr parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 599becb359ff15356a925985d03612b7 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4699e5ce-70e7-4911-84c2-08ba672c9e11_isrForCrosstalkSources_1031706_19.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4699e5ce-70e7-4911-84c2-08ba672c9e11_isrForCrosstalkSources_1031706_19.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash cd74dca1e51106bbfbcc358c9c5ab795 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/072719b1-30ba-4e6a-81f5-60d73952935e_isrForCrosstalkSources_1031719_35.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/072719b1-30ba-4e6a-81f5-60d73952935e_isrForCrosstalkSources_1031719_35.stderr parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 5f48a2e12599d5a9359d178585503865 parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b3a55dd0-7dbf-401c-9a28-a92577fda5a6_isrForCrosstalkSources_1031708_37.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b3a55dd0-7dbf-401c-9a28-a92577fda5a6_isrForCrosstalkSources_1031708_37.stderr parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 22b3778195d4360f027f464f30655480 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7f02ce67-0da2-43b1-ab1c-28b26123a90c_isrForCrosstalkSources_1031721_49.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7f02ce67-0da2-43b1-ab1c-28b26123a90c_isrForCrosstalkSources_1031721_49.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 8b6893e533af5f01eea31fd1d07c3721 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a1ccae7d-e93f-4d0c-b205-866d9e62be37_isrForCrosstalkSources_1031718_45.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a1ccae7d-e93f-4d0c-b205-866d9e62be37_isrForCrosstalkSources_1031718_45.stderr parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash 3424115bfc0154ab84bea18e8567f901 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/e9098657-29d2-419f-a1dc-4da55d489999_isrForCrosstalkSources_1031737_46.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/e9098657-29d2-419f-a1dc-4da55d489999_isrForCrosstalkSources_1031737_46.stderr parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 99e3d4ab5a922d88726857af19f80dd9 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ec97510c-af9e-4266-8c3b-66a5d6349363_isrForCrosstalkSources_1031738_16.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ec97510c-af9e-4266-8c3b-66a5d6349363_isrForCrosstalkSources_1031738_16.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 3ecda23bf51c77ebc447a9fd53f9d800 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/19a58c1c-6a36-49c6-9ecf-16812b0d5899_isrForCrosstalkSources_1031715_60.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/19a58c1c-6a36-49c6-9ecf-16812b0d5899_isrForCrosstalkSources_1031715_60.stderr parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash b499802050128191eccfee2bc19f0a03 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/02fa0833-3fe8-46e3-8ebc-efaad8076fbc_isrForCrosstalkSources_1031726_7.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/02fa0833-3fe8-46e3-8ebc-efaad8076fbc_isrForCrosstalkSources_1031726_7.stderr parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash b9f1d022735e1a49f21cd229d6fdc2a4 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/88469836-5683-427e-80d6-cd05f50612c4_isrForCrosstalkSources_1031722_25.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/88469836-5683-427e-80d6-cd05f50612c4_isrForCrosstalkSources_1031722_25.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash c7712dfc78e3d84b415112a5bcc7759b parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/24c8cd32-85f1-4f4c-989e-5eff645aec73_isrForCrosstalkSources_1031719_3.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/24c8cd32-85f1-4f4c-989e-5eff645aec73_isrForCrosstalkSources_1031719_3.stderr parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 71c6a1054e7429804d2bd006fabef7bb parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ef69c233-6ed4-455e-ba8e-a61705e75bb2_isrForCrosstalkSources_1031717_56.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ef69c233-6ed4-455e-ba8e-a61705e75bb2_isrForCrosstalkSources_1031717_56.stderr parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash ccb6bb106966382da7675d4dde7d5533 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/1085ba3b-f4f3-4470-99d4-df84f24b435e_isrForCrosstalkSources_1031707_28.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/1085ba3b-f4f3-4470-99d4-df84f24b435e_isrForCrosstalkSources_1031707_28.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash ef06322f36448852fbc40f7af86fbd9a parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e9b9bc7f-45d9-47b4-9e76-ec28f8c00908_isrForCrosstalkSources_1031716_31.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e9b9bc7f-45d9-47b4-9e76-ec28f8c00908_isrForCrosstalkSources_1031716_31.stderr parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 6e72e66473e823a52b16f43eec3ee139 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/5b19da89-caa0-4dbe-8593-e8bdf9cc98e3_isrForCrosstalkSources_1031713_35.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/5b19da89-caa0-4dbe-8593-e8bdf9cc98e3_isrForCrosstalkSources_1031713_35.stderr parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 692d18f0761a6bc718801b80a1d93469 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/562a4569-b67e-4428-910d-ba819ac56593_isrForCrosstalkSources_1031699_4.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/562a4569-b67e-4428-910d-ba819ac56593_isrForCrosstalkSources_1031699_4.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash f8b7876b5706262975eaf6ed9bd41fe0 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a250396b-070c-4e46-a0d9-2f6e23c31492_isrForCrosstalkSources_1031736_22.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a250396b-070c-4e46-a0d9-2f6e23c31492_isrForCrosstalkSources_1031736_22.stderr parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash bc6049d818dc7135a5ee2cbe320b501c parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f3cdf290-e1d6-4bd3-971f-ea7fc592c814_isrForCrosstalkSources_1031709_2.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f3cdf290-e1d6-4bd3-971f-ea7fc592c814_isrForCrosstalkSources_1031709_2.stderr parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash c3d8aeb7778ffd2e99c11fc27c071835 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/75d8cbf0-6d39-4b41-978d-5f30c84c2cc6_isrForCrosstalkSources_1031702_9.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/75d8cbf0-6d39-4b41-978d-5f30c84c2cc6_isrForCrosstalkSources_1031702_9.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 96a08f962bc86164e14ddf631cf0ea4b parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e866cba7-ba0b-4fee-9d5e-1c696b81c748_isrForCrosstalkSources_1031733_17.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e866cba7-ba0b-4fee-9d5e-1c696b81c748_isrForCrosstalkSources_1031733_17.stderr parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 22d55a6d2caff7e9aa9806cffa52865c parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/41b0f51c-4ed7-45b3-9053-8768712bf953_isrForCrosstalkSources_1031711_45.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/41b0f51c-4ed7-45b3-9053-8768712bf953_isrForCrosstalkSources_1031711_45.stderr parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash 4406712d49fa6cc4a16dc08cc47473a1 parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/6ba78d59-5298-4eb8-aa6e-9e5859bd112b_isrForCrosstalkSources_1031712_44.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/6ba78d59-5298-4eb8-aa6e-9e5859bd112b_isrForCrosstalkSources_1031712_44.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 2db47fd84b39cd1b193a064b021a2d95 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c535190b-69c0-48ea-92eb-f5e7e0db1f8a_isrForCrosstalkSources_1031734_14.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c535190b-69c0-48ea-92eb-f5e7e0db1f8a_isrForCrosstalkSources_1031734_14.stderr parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 0870d8ca6971d5240223dcc2755a561b parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:04 proc_lsst.multi:170 [INFO] [multi] job_id 17147938 proc_lsst.multi INFO: [multi] job_id 17147938 2024-03-17 04:02:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 04:02:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147938 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147938 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17147938 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-17 04:02:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-17 04:02:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 04:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/5053c429-107d-49d7-979a-949d1dbeb840_isrForCrosstalkSources_1031710_8.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/5053c429-107d-49d7-979a-949d1dbeb840_isrForCrosstalkSources_1031710_8.stderr parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash dcef725091cb7ebdaada7fe8184c47d6 parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0521acf4-0607-4058-bcfb-9b172579c44a_isrForCrosstalkSources_1031705_18.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0521acf4-0607-4058-bcfb-9b172579c44a_isrForCrosstalkSources_1031705_18.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash ca22ea2d2e0ae931def6a5642b969794 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/15a231dc-e121-44e2-80d7-b5b551093ff9_isrForCrosstalkSources_1031711_32.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/15a231dc-e121-44e2-80d7-b5b551093ff9_isrForCrosstalkSources_1031711_32.stderr parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash c0771fb3c5630b8459b6fd777f8f74ce parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/43da2c44-de65-481a-871b-f65fcd39a62b_isrForCrosstalkSources_1031736_54.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/43da2c44-de65-481a-871b-f65fcd39a62b_isrForCrosstalkSources_1031736_54.stderr parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash bd819119c77030c7a643fa615cd09a8a parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/47df78b0-2e3b-4f88-b68c-4fd73a600ac0_isrForCrosstalkSources_1031699_5.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/47df78b0-2e3b-4f88-b68c-4fd73a600ac0_isrForCrosstalkSources_1031699_5.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash 74d9c59f6f8f9bd7210edfaa7c0b52be parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ae7b22ec-baac-46e4-9667-351857230fc8_isrForCrosstalkSources_1031737_49.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ae7b22ec-baac-46e4-9667-351857230fc8_isrForCrosstalkSources_1031737_49.stderr parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 8c9bb89d9af9e6863b12f85d89872672 parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3eaf4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a9df5cb2-bd44-4dca-8fed-30e8aaa3589b_isrForCrosstalkSources_1031736_62.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a9df5cb2-bd44-4dca-8fed-30e8aaa3589b_isrForCrosstalkSources_1031736_62.stderr parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash b278220a1e3c343256426ccebee6d8bc parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/b3571dfc-e4f8-4eeb-8d57-89ce4090af92_isrForCrosstalkSources_1031731_55.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/b3571dfc-e4f8-4eeb-8d57-89ce4090af92_isrForCrosstalkSources_1031731_55.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 0f24bac24ef0d1e14df1443db98966e1 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/d7ead2b4-b700-43bb-9ee5-44a7a73b6909_isrForCrosstalkSources_1031700_29.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/d7ead2b4-b700-43bb-9ee5-44a7a73b6909_isrForCrosstalkSources_1031700_29.stderr parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 6fa5849148ae49d79982a53418c8ac97 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/98878813-26c7-4941-98bf-919150677cee_isrForCrosstalkSources_1031725_47.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/98878813-26c7-4941-98bf-919150677cee_isrForCrosstalkSources_1031725_47.stderr parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash cb0711a9b67d9c55dc2d3bb6b3f0d6c7 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/2c451775-3d48-4ca7-9436-76e262c661de_isrForCrosstalkSources_1031700_58.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/2c451775-3d48-4ca7-9436-76e262c661de_isrForCrosstalkSources_1031700_58.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash f6b6f9b680f8aaeff259306cfbadd384 parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/e2f2697f-7783-4705-9047-537713726ec0_isrForCrosstalkSources_1031715_25.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/e2f2697f-7783-4705-9047-537713726ec0_isrForCrosstalkSources_1031715_25.stderr parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 9dbf7673655c32a1322ea0ed3b689986 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8d76f3e4-4b2c-4588-ae24-bd425351bac1_isrForCrosstalkSources_1031707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8d76f3e4-4b2c-4588-ae24-bd425351bac1_isrForCrosstalkSources_1031707_2.stderr parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 6cb490cba0c17443ff1b6edefad66b29 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a045939d-1940-4381-a3ba-0479cb34caf9_isrForCrosstalkSources_1031722_13.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a045939d-1940-4381-a3ba-0479cb34caf9_isrForCrosstalkSources_1031722_13.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash db07f944a41279c9ad578217a983b638 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/6da9d678-0922-4aff-80ef-5c9188f9cf19_isrForCrosstalkSources_1031703_28.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/6da9d678-0922-4aff-80ef-5c9188f9cf19_isrForCrosstalkSources_1031703_28.stderr parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 621a24eccf90f1d3a073a0a6051edda8 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/98fb536a-8880-44e8-ab58-b7e914f01fc6_isrForCrosstalkSources_1031716_11.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/98fb536a-8880-44e8-ab58-b7e914f01fc6_isrForCrosstalkSources_1031716_11.stderr parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash bbfa2a459a2ca2a69502c63173bebc68 parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/bfd17d9c-5498-4faa-af1f-1c7de738e9ad_isrForCrosstalkSources_1031737_30.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/bfd17d9c-5498-4faa-af1f-1c7de738e9ad_isrForCrosstalkSources_1031737_30.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash aa96db6639854301f866de816f226029 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/89359499-62ca-48ba-af00-89580dac413e_isrForCrosstalkSources_1031730_38.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/89359499-62ca-48ba-af00-89580dac413e_isrForCrosstalkSources_1031730_38.stderr parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash d8310487d7fb09762236683033da4240 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/0485c8f5-40ae-4f17-8f1f-d823fbd4ee79_isrForCrosstalkSources_1031702_32.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/0485c8f5-40ae-4f17-8f1f-d823fbd4ee79_isrForCrosstalkSources_1031702_32.stderr parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 5e57aea93fe3ce96aa911b23a3211436 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/3ebc3330-4efc-427d-9bb5-c0caf72d333b_isrForCrosstalkSources_1031727_52.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/3ebc3330-4efc-427d-9bb5-c0caf72d333b_isrForCrosstalkSources_1031727_52.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash f5eacdbf9a3d80f889774cc105ec12b9 parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/d58e91f0-05c3-416e-8e34-432564f45b92_isrForCrosstalkSources_1031721_48.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/d58e91f0-05c3-416e-8e34-432564f45b92_isrForCrosstalkSources_1031721_48.stderr parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash b50ac2a1c271e2e3f8bf877829588b44 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/837f3e37-e753-412f-ada8-8e89f2a59da2_isrForCrosstalkSources_1031704_46.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/837f3e37-e753-412f-ada8-8e89f2a59da2_isrForCrosstalkSources_1031704_46.stderr parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 97164b37b2e92946d7576059fcd1450e parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ea3f8cc6-2c4f-4ef9-adf5-5bdc1ff06f39_isrForCrosstalkSources_1031713_47.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ea3f8cc6-2c4f-4ef9-adf5-5bdc1ff06f39_isrForCrosstalkSources_1031713_47.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 09204e6107a5c46a3f1e9d70d198744e parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/12eac9b1-b3e8-4658-b0ce-6f5eff36b476_isrForCrosstalkSources_1031723_42.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/12eac9b1-b3e8-4658-b0ce-6f5eff36b476_isrForCrosstalkSources_1031723_42.stderr parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 8c6e8dfab812cfaf8a68a71bfe7f6070 parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/638f50df-48bb-4935-9535-e007459f7d7e_isrForCrosstalkSources_1031732_48.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/638f50df-48bb-4935-9535-e007459f7d7e_isrForCrosstalkSources_1031732_48.stderr parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 9965ea5f5eca958aefa9a81e6ea62587 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5a72814e-269d-4009-8958-42c6a09306e1_isrForCrosstalkSources_1031720_44.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5a72814e-269d-4009-8958-42c6a09306e1_isrForCrosstalkSources_1031720_44.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash d5a8b52f5aa2e780b1db219308127c32 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/665cca77-ec3b-432f-9678-5d9ea80dadaa_isrForCrosstalkSources_1031705_29.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/665cca77-ec3b-432f-9678-5d9ea80dadaa_isrForCrosstalkSources_1031705_29.stderr parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 4301aad71e4417ec4e30526653314649 parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a545e35e-958c-4fdb-a519-d2fc21eca840_isrForCrosstalkSources_1031706_60.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a545e35e-958c-4fdb-a519-d2fc21eca840_isrForCrosstalkSources_1031706_60.stderr parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 03ad687d38d61a3b9bc1d8ed83ed040f parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/099dad65-b159-4f37-b3f8-d71a1bac4d84_isrForCrosstalkSources_1031725_25.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/099dad65-b159-4f37-b3f8-d71a1bac4d84_isrForCrosstalkSources_1031725_25.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 5bfcb62f0d00126d6b2e11e7de155451 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/e1190e8a-21ea-4b31-b333-a0f1ea8b6d83_isrForCrosstalkSources_1031711_52.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/e1190e8a-21ea-4b31-b333-a0f1ea8b6d83_isrForCrosstalkSources_1031711_52.stderr parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash a2a09b9ddfae36f1682f109fb78144ee parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8f79e1ff-44a4-475d-a35d-d3d0c466ec39_isrForCrosstalkSources_1031709_21.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8f79e1ff-44a4-475d-a35d-d3d0c466ec39_isrForCrosstalkSources_1031709_21.stderr parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 79c57a9b9a7e255471c871ec1a834c60 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0172a12b-84e8-4f47-b4a3-4938ca4c2d0c_isrForCrosstalkSources_1031711_2.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0172a12b-84e8-4f47-b4a3-4938ca4c2d0c_isrForCrosstalkSources_1031711_2.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash b365449e9489ed6346fc72b2b832bb45 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9d42aee9-585c-4fe6-bf87-14a92a1727ef_isrForCrosstalkSources_1031733_37.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9d42aee9-585c-4fe6-bf87-14a92a1727ef_isrForCrosstalkSources_1031733_37.stderr parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 092414497ecefb1fcd570a75348ec66d parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/971a7ad1-44d4-4725-b516-4f993e7e8e01_isrForCrosstalkSources_1031722_15.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/971a7ad1-44d4-4725-b516-4f993e7e8e01_isrForCrosstalkSources_1031722_15.stderr parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 8a0f23d157e56e077e99c53dfa4b5741 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b6a80008-2956-417c-8936-600ec0affb43_isrForCrosstalkSources_1031714_9.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b6a80008-2956-417c-8936-600ec0affb43_isrForCrosstalkSources_1031714_9.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash aacc89116334bac733e871fa643db1c5 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f041b2fc-2ea5-400d-a5ac-386d45cfa6af_isrForCrosstalkSources_1031716_42.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f041b2fc-2ea5-400d-a5ac-386d45cfa6af_isrForCrosstalkSources_1031716_42.stderr parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash bbcc6df33a92b3a055e9b6d14222a352 parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4deed72c-ba8a-49cc-b345-8d5143bd78e3_isrForCrosstalkSources_1031718_32.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4deed72c-ba8a-49cc-b345-8d5143bd78e3_isrForCrosstalkSources_1031718_32.stderr parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 2518a48be200879c9a3cfa53a9d04476 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/74de8857-7d00-4d13-9080-d16d07c81ea9_isrForCrosstalkSources_1031712_15.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/74de8857-7d00-4d13-9080-d16d07c81ea9_isrForCrosstalkSources_1031712_15.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 9ff0ab9935899c3151895fe0cc242ed6 parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b5eeae9e-81bd-4088-b568-295019b602f4_isrForCrosstalkSources_1031718_2.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b5eeae9e-81bd-4088-b568-295019b602f4_isrForCrosstalkSources_1031718_2.stderr parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 86776c9eea0d08f970d7f11eb2cf7a10 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7b2f3f29-2b4b-4ef2-94bb-57800e76afb4_isrForCrosstalkSources_1031721_39.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7b2f3f29-2b4b-4ef2-94bb-57800e76afb4_isrForCrosstalkSources_1031721_39.stderr parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash d2b52028d0a798bc89b5b5f1e9c0a724 parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/c745b8bd-0a54-4d9e-96eb-ce814b035a41_isrForCrosstalkSources_1031713_51.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/c745b8bd-0a54-4d9e-96eb-ce814b035a41_isrForCrosstalkSources_1031713_51.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash ff8234a61af732fce0d327ab8955c729 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3a5e41bb-5e6b-493b-8636-3017ab319d80_isrForCrosstalkSources_1031736_24.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3a5e41bb-5e6b-493b-8636-3017ab319d80_isrForCrosstalkSources_1031736_24.stderr parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash d2983ec95ba08df01340b8a20c300542 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/be5ee021-ae19-4c1d-a59e-ed1ca99ba651_isrForCrosstalkSources_1031733_4.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/be5ee021-ae19-4c1d-a59e-ed1ca99ba651_isrForCrosstalkSources_1031733_4.stderr parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 53f9a2c5e378e836c7f7c449f92b9457 parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/42ea1df2-0399-497e-84aa-bd7ee80c1b6a_isrForCrosstalkSources_1031727_16.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/42ea1df2-0399-497e-84aa-bd7ee80c1b6a_isrForCrosstalkSources_1031727_16.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash 5a9b25d89055696c05ce894946b4ac95 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/aa9dd174-6339-4e26-8060-0b45d487b614_isrForCrosstalkSources_1031729_8.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/aa9dd174-6339-4e26-8060-0b45d487b614_isrForCrosstalkSources_1031729_8.stderr parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 1807484cb8acd8c42f9781b3864ce021 parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/73af8567-f9ae-401c-a115-9ad796e60674_isrForCrosstalkSources_1031717_36.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/73af8567-f9ae-401c-a115-9ad796e60674_isrForCrosstalkSources_1031717_36.stderr parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 85f1e3f8bd51639337bffff2fe580267 parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/cf01b3af-dcb4-4893-8ece-cd9e242302a6_isrForCrosstalkSources_1031719_59.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/cf01b3af-dcb4-4893-8ece-cd9e242302a6_isrForCrosstalkSources_1031719_59.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 5c4bdd482e8de67ba0860d026cb8adf4 parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c2ea52d6-231e-446b-8c51-54b7d61883ea_isrForCrosstalkSources_1031712_45.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c2ea52d6-231e-446b-8c51-54b7d61883ea_isrForCrosstalkSources_1031712_45.stderr parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash cb6d01b21283f10ad9a3165c0ff01ec3 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ac093d7-a4bc-4f3e-b5a5-ec55aec84d80_isrForCrosstalkSources_1031719_43.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ac093d7-a4bc-4f3e-b5a5-ec55aec84d80_isrForCrosstalkSources_1031719_43.stderr parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 4a489cfb432f16167477694d1537bec7 parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a8998a46-dbac-4769-ad4f-18dc9be86c10_isrForCrosstalkSources_1031726_36.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a8998a46-dbac-4769-ad4f-18dc9be86c10_isrForCrosstalkSources_1031726_36.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 58f172edcbf7ff5dc816dce46c25d08a parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/24a6ccd2-f1ff-4846-bdec-b37d187ea54b_isrForCrosstalkSources_1031726_44.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/24a6ccd2-f1ff-4846-bdec-b37d187ea54b_isrForCrosstalkSources_1031726_44.stderr parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash 41256f712571698ecb905a38403ae55a parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/eeef0511-df72-4060-a06a-224f05dd244b_isrForCrosstalkSources_1031716_3.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/eeef0511-df72-4060-a06a-224f05dd244b_isrForCrosstalkSources_1031716_3.stderr parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 069bb50ed24d1f6c42ad1c1a7215c6e1 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cec8b751-0217-4d2b-b126-23ad55bbfdfb_isrForCrosstalkSources_1031728_27.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cec8b751-0217-4d2b-b126-23ad55bbfdfb_isrForCrosstalkSources_1031728_27.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash e41b837b72fd8014e623a14ed7beb4ec parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/16cbf79c-b44d-442f-832e-96bd63f9a611_isrForCrosstalkSources_1031732_38.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/16cbf79c-b44d-442f-832e-96bd63f9a611_isrForCrosstalkSources_1031732_38.stderr parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 0c7bbdfc8e9ccb7bdd36a3acb2fc217a parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cdc2bbe8-828e-4baf-8995-e42ef5ee29e3_isrForCrosstalkSources_1031716_50.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cdc2bbe8-828e-4baf-8995-e42ef5ee29e3_isrForCrosstalkSources_1031716_50.stderr parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash dfc017b7b004cccada5929e3ae1ad122 parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/bd6615e7-6b46-429e-b801-304d1c8ad270_isrForCrosstalkSources_1031700_40.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/bd6615e7-6b46-429e-b801-304d1c8ad270_isrForCrosstalkSources_1031700_40.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash d5e1abcf4bb28859b0fa4f8ba4a7430c parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/365f87a6-dae5-4349-90ef-9575bee7ad3c_isrForCrosstalkSources_1031727_38.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/365f87a6-dae5-4349-90ef-9575bee7ad3c_isrForCrosstalkSources_1031727_38.stderr parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 7827d8937a9aaea2b15a33e4976ce4c8 parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7fe46a34-796b-4fda-90ab-affe1b6aa8a3_isrForCrosstalkSources_1031702_40.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7fe46a34-796b-4fda-90ab-affe1b6aa8a3_isrForCrosstalkSources_1031702_40.stderr parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash ba1e8d54de4866657185ad322eb6881b parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/5c7c0c9b-c633-4059-a028-03d2777164b5_isrForCrosstalkSources_1031723_10.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/5c7c0c9b-c633-4059-a028-03d2777164b5_isrForCrosstalkSources_1031723_10.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash c7314e7d999647dc1c2409f5c482067e parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/6d1730f5-9f47-4ff1-afa8-dfd7056467f2_isrForCrosstalkSources_1031735_13.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/6d1730f5-9f47-4ff1-afa8-dfd7056467f2_isrForCrosstalkSources_1031735_13.stderr parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 8ae2c6942afb4972ff45a2b100b3e820 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1c55d933-c7a1-4826-ba91-398ab361666d_isrForCrosstalkSources_1031706_20.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1c55d933-c7a1-4826-ba91-398ab361666d_isrForCrosstalkSources_1031706_20.stderr parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash c762a59c047b6797c55d8b4b23435000 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8bc60305-a80d-45d3-b1bd-4e75c60118ea_isrForCrosstalkSources_1031719_47.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8bc60305-a80d-45d3-b1bd-4e75c60118ea_isrForCrosstalkSources_1031719_47.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash ea059e5bc28b95892f925a6876606f1e parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6ea3ee8c-c9b1-424b-96e9-39fa02f8153d_isrForCrosstalkSources_1031728_26.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6ea3ee8c-c9b1-424b-96e9-39fa02f8153d_isrForCrosstalkSources_1031728_26.stderr parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash c173f12f8d5c428eb073820beceec05d parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/0f7c0dc8-f619-423f-bdae-57cf4ff3b5b5_isrForCrosstalkSources_1031729_44.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/0f7c0dc8-f619-423f-bdae-57cf4ff3b5b5_isrForCrosstalkSources_1031729_44.stderr parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash c1bf0e3dce7a835b74f81e7c15a1ba7d parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9a570f92-c71e-4ad5-84ec-53287165f0e8_isrForCrosstalkSources_1031730_24.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9a570f92-c71e-4ad5-84ec-53287165f0e8_isrForCrosstalkSources_1031730_24.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash e3f17032815c47bac35b304aaeaaaf93 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/1fd6e26e-6ae9-4af7-85cb-57b11a788ff7_isrForCrosstalkSources_1031717_8.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/1fd6e26e-6ae9-4af7-85cb-57b11a788ff7_isrForCrosstalkSources_1031717_8.stderr parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash a867ba007130cc1fffc6d2e06c2aa511 parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d3b960f2-a895-481d-b87a-46eafb3cc69f_isrForCrosstalkSources_1031731_19.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d3b960f2-a895-481d-b87a-46eafb3cc69f_isrForCrosstalkSources_1031731_19.stderr parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash d833e6fcd18e2520ef3e331e3870696c parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/85dabb50-5845-4ec0-9e19-e426623a119e_isrForCrosstalkSources_1031702_57.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/85dabb50-5845-4ec0-9e19-e426623a119e_isrForCrosstalkSources_1031702_57.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash b60bff48bfc154180a9e2304e9cbfc6b parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/dccc054c-81f7-4450-aeac-cec6d907a509_isrForCrosstalkSources_1031711_57.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/dccc054c-81f7-4450-aeac-cec6d907a509_isrForCrosstalkSources_1031711_57.stderr parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash 9e5e5649d3d4f9c2ac717ede8a177af0 parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2a0f0cfa-fc7d-485b-875e-fd2228a144e0_isrForCrosstalkSources_1031726_31.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2a0f0cfa-fc7d-485b-875e-fd2228a144e0_isrForCrosstalkSources_1031726_31.stderr parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 090ada0bc44fe56be50a5e0a7ecd38c9 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/6990de19-5752-4a41-ad41-096a7bd5d5a1_isrForCrosstalkSources_1031719_46.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/6990de19-5752-4a41-ad41-096a7bd5d5a1_isrForCrosstalkSources_1031719_46.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 79910d843b311649cebc16d4accdced8 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9271b7bd-9f4f-41c3-9415-b1b97da5598b_isrForCrosstalkSources_1031729_31.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9271b7bd-9f4f-41c3-9415-b1b97da5598b_isrForCrosstalkSources_1031729_31.stderr parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 739fe30a38eef4f2f6e69702f9c567a8 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a48a9dc0-e4d7-4c36-8ba1-14076e9565ff_isrForCrosstalkSources_1031709_50.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a48a9dc0-e4d7-4c36-8ba1-14076e9565ff_isrForCrosstalkSources_1031709_50.stderr parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash fadc2a715fd65ee409d2b83b36c3bf5c parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ccd86a5a-19b5-4b40-ae28-d7ad56f86da6_isrForCrosstalkSources_1031725_18.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ccd86a5a-19b5-4b40-ae28-d7ad56f86da6_isrForCrosstalkSources_1031725_18.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 50bc0661b61683426416d48f27b03075 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3c04595b-bffe-4c03-9a37-222a7f2aa7bd_isrForCrosstalkSources_1031719_32.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3c04595b-bffe-4c03-9a37-222a7f2aa7bd_isrForCrosstalkSources_1031719_32.stderr parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 5f904d0853bbca93030ceaddd13150f0 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/20cd5d15-7d36-4ee3-b031-421527b150c5_isrForCrosstalkSources_1031700_39.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/20cd5d15-7d36-4ee3-b031-421527b150c5_isrForCrosstalkSources_1031700_39.stderr parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 2e4c58906fe42fa42b9f790234289672 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ab217f0-9b21-4f87-a76c-835dfd6a4bb0_isrForCrosstalkSources_1031719_49.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ab217f0-9b21-4f87-a76c-835dfd6a4bb0_isrForCrosstalkSources_1031719_49.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash 7f242e46c385b3df50446ef7738ca068 parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/e3d884da-9dcc-4c8e-bffb-6ae24af88341_isrForCrosstalkSources_1031715_9.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/e3d884da-9dcc-4c8e-bffb-6ae24af88341_isrForCrosstalkSources_1031715_9.stderr parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash ed40d6890aeabaaad9fd50cbf685a2f3 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/5c1eac5b-bff8-45e3-9c9e-f4d03bd7f8be_isrForCrosstalkSources_1031699_60.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/5c1eac5b-bff8-45e3-9c9e-f4d03bd7f8be_isrForCrosstalkSources_1031699_60.stderr parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 4c4903e2c9143e4ad2471953063093ed parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1814685b-0861-4df4-a127-b750e5a29845_isrForCrosstalkSources_1031700_50.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1814685b-0861-4df4-a127-b750e5a29845_isrForCrosstalkSources_1031700_50.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash a394671f4b064dbedf899f688b84215e parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:04 proc_lsst.multi:170 [INFO] [multi] job_id 17147939 proc_lsst.multi INFO: [multi] job_id 17147939 2024-03-17 04:02:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 04:02:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147939 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147939 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17147939 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-17 04:02:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-17 04:02:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 04:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b90831c0-5887-4e95-8c8c-d2a0c414931c_isrForCrosstalkSources_1031728_60.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b90831c0-5887-4e95-8c8c-d2a0c414931c_isrForCrosstalkSources_1031728_60.stderr parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 03e52b1f873a536ebce17cf2ad717bd1 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3505b4c5-bd5e-433b-b8e8-b1372e8ace16_isrForCrosstalkSources_1031738_11.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3505b4c5-bd5e-433b-b8e8-b1372e8ace16_isrForCrosstalkSources_1031738_11.stderr parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 286c5a322ebc7607fad3b6ded3ad9c6e parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/fdc6cae2-8f7c-4020-8094-b61ecd4266cc_isrForCrosstalkSources_1031699_35.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/fdc6cae2-8f7c-4020-8094-b61ecd4266cc_isrForCrosstalkSources_1031699_35.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash 67097b487acb58b5af7c5c7b3a2df117 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0a70c4ca-3a9b-45df-a343-093e9131d91a_isrForCrosstalkSources_1031705_22.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0a70c4ca-3a9b-45df-a343-093e9131d91a_isrForCrosstalkSources_1031705_22.stderr parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 7f29ffc29c37d8fa10abb856d64d57bd parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/78be94f4-0a28-466b-843b-04f69a4ab4b3_isrForCrosstalkSources_1031738_5.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/78be94f4-0a28-466b-843b-04f69a4ab4b3_isrForCrosstalkSources_1031738_5.stderr parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash ad24b79d9fb0d8228156461c1c00debf parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/963f76c1-f67c-4351-a161-87638ac92d80_isrForCrosstalkSources_1031699_39.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/963f76c1-f67c-4351-a161-87638ac92d80_isrForCrosstalkSources_1031699_39.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash c600952f0b9c2a5576a9c5d332bfdab6 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1021fb94-6e85-47a0-88d6-f400f7026b2c_isrForCrosstalkSources_1031735_25.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1021fb94-6e85-47a0-88d6-f400f7026b2c_isrForCrosstalkSources_1031735_25.stderr parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 7c849c50bb910cba3ca4ba70a956b569 parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/944d4a2a-d675-4134-ae58-3689a48a7e6d_isrForCrosstalkSources_1031724_19.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/944d4a2a-d675-4134-ae58-3689a48a7e6d_isrForCrosstalkSources_1031724_19.stderr parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 5ad473c93a0ee94ed9fdaf42375f8ee1 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b23fc4e9-11e2-4951-967e-5ae7856dfd82_isrForCrosstalkSources_1031707_43.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b23fc4e9-11e2-4951-967e-5ae7856dfd82_isrForCrosstalkSources_1031707_43.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 6b3240e4181f938828ec6eb788033ae9 parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/211be384-be48-4516-9338-cffa6d07df85_isrForCrosstalkSources_1031702_38.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/211be384-be48-4516-9338-cffa6d07df85_isrForCrosstalkSources_1031702_38.stderr parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 5e0f04b6da4e4302592f09453100b7bd parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f489b3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d196ef69-f6fa-40e3-9f45-236e8cc5c0f1_isrForCrosstalkSources_1031728_10.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d196ef69-f6fa-40e3-9f45-236e8cc5c0f1_isrForCrosstalkSources_1031728_10.stderr parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 8a03de97c1ab1d070a389fa3430ee92e parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c56669a4-d18e-4543-8158-a8eafd2b5dfb_isrForCrosstalkSources_1031710_9.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c56669a4-d18e-4543-8158-a8eafd2b5dfb_isrForCrosstalkSources_1031710_9.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 89ecdb485fd5926889b366dc4add9833 parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c72ebb1e-0a34-412a-aa10-d63cd3321ec2_isrForCrosstalkSources_1031728_54.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c72ebb1e-0a34-412a-aa10-d63cd3321ec2_isrForCrosstalkSources_1031728_54.stderr parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 6f3fb1ea59c38a16e0a95658d7035eff parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a8d41e6e-c2bf-46ea-9478-e9bd95eb70cf_isrForCrosstalkSources_1031721_57.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a8d41e6e-c2bf-46ea-9478-e9bd95eb70cf_isrForCrosstalkSources_1031721_57.stderr parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 256e49243eb732d9816953adbc8eda0a parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/26b6e16e-d899-4afb-9a8d-3d1ba4f0906c_isrForCrosstalkSources_1031727_5.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/26b6e16e-d899-4afb-9a8d-3d1ba4f0906c_isrForCrosstalkSources_1031727_5.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash edee339094a2ce6649b3b7fcd79523c6 parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0eae4419-4608-43e8-bc5c-db6611a1490a_isrForCrosstalkSources_1031712_27.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0eae4419-4608-43e8-bc5c-db6611a1490a_isrForCrosstalkSources_1031712_27.stderr parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 6dd63054fb28c2ee4985152b980b9df4 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d69b9c09-eedf-451a-bb63-27a591901f4f_isrForCrosstalkSources_1031733_1.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d69b9c09-eedf-451a-bb63-27a591901f4f_isrForCrosstalkSources_1031733_1.stderr parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash dda3397b105e7e224febead7c5ea655f parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e2012346-8d63-467c-93bb-81b9f93d1440_isrForCrosstalkSources_1031706_47.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e2012346-8d63-467c-93bb-81b9f93d1440_isrForCrosstalkSources_1031706_47.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 7aee8aeba161bbba6d8107c15968b3c4 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c4cce60d-98a5-4173-b8fe-eb14e2e1bc97_isrForCrosstalkSources_1031702_4.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c4cce60d-98a5-4173-b8fe-eb14e2e1bc97_isrForCrosstalkSources_1031702_4.stderr parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 2cfe9b90772520b75f280fc99f283069 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2f47466c-1e5d-48cf-9480-418cbb34a1ea_isrForCrosstalkSources_1031709_23.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2f47466c-1e5d-48cf-9480-418cbb34a1ea_isrForCrosstalkSources_1031709_23.stderr parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash 79e260b6b6ac572d015aaa569f372f4d parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/7ba70e4e-e56f-4373-b9df-e59f40db0cba_isrForCrosstalkSources_1031715_58.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/7ba70e4e-e56f-4373-b9df-e59f40db0cba_isrForCrosstalkSources_1031715_58.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash d61db018d87517b8563983fd38000545 parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f2ed30fa-ea4c-4e14-9962-bc40935db797_isrForCrosstalkSources_1031709_46.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f2ed30fa-ea4c-4e14-9962-bc40935db797_isrForCrosstalkSources_1031709_46.stderr parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 6b4bb4423a75458742b988d395b390e6 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/aad5a357-1a3f-4aa9-85c1-0718e0463303_isrForCrosstalkSources_1031719_36.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/aad5a357-1a3f-4aa9-85c1-0718e0463303_isrForCrosstalkSources_1031719_36.stderr parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash c060735e3b256736b6cab5d6da38b484 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e08b847e-20b3-4228-8303-a3716be336fe_isrForCrosstalkSources_1031703_20.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e08b847e-20b3-4228-8303-a3716be336fe_isrForCrosstalkSources_1031703_20.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash f577addf3015526a38433f8a87e476db parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6a41316c-5c15-41f8-910b-7b9ade84d89c_isrForCrosstalkSources_1031726_17.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6a41316c-5c15-41f8-910b-7b9ade84d89c_isrForCrosstalkSources_1031726_17.stderr parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 0bcdc5512400cc677fe6edee3efc60ac parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4b5f9846-d165-4e37-b1a6-88a6221fd6a8_isrForCrosstalkSources_1031738_36.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4b5f9846-d165-4e37-b1a6-88a6221fd6a8_isrForCrosstalkSources_1031738_36.stderr parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 728abcedd5124a3406f6de3e3141d08a parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/1112a384-6567-492c-ae5d-e7cef503df14_isrForCrosstalkSources_1031702_21.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/1112a384-6567-492c-ae5d-e7cef503df14_isrForCrosstalkSources_1031702_21.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 209076351ac61a107000cede699fed68 parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f9cc3402-09c8-4caa-a53b-f99f28b442ea_isrForCrosstalkSources_1031710_45.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f9cc3402-09c8-4caa-a53b-f99f28b442ea_isrForCrosstalkSources_1031710_45.stderr parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 81edf8ea738f5514c2a0b991e58f53b5 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/ec953354-1016-4430-a8c4-df63a00e46bd_isrForCrosstalkSources_1031707_56.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/ec953354-1016-4430-a8c4-df63a00e46bd_isrForCrosstalkSources_1031707_56.stderr parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 7c329a8266d3891ea25d0ac6e18fa98e parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c6ee71f7-6fc4-4422-a041-227dc2cfa73e_isrForCrosstalkSources_1031706_52.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c6ee71f7-6fc4-4422-a041-227dc2cfa73e_isrForCrosstalkSources_1031706_52.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash d24abee8be7ab39d1c85a7a702888682 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/da6e23ae-dc7f-4c43-a23e-290af477c632_isrForCrosstalkSources_1031733_19.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/da6e23ae-dc7f-4c43-a23e-290af477c632_isrForCrosstalkSources_1031733_19.stderr parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash f58a7bee3944b30cc8580815012f16ae parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/50c87e5f-55ed-4422-8e29-d86ecbc50763_isrForCrosstalkSources_1031733_35.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/50c87e5f-55ed-4422-8e29-d86ecbc50763_isrForCrosstalkSources_1031733_35.stderr parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash dcc9ec2d5b399db59fd090de87345a8e parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/49cafed2-3169-4dd6-83f3-1ed8561cf9c5_isrForCrosstalkSources_1031704_19.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/49cafed2-3169-4dd6-83f3-1ed8561cf9c5_isrForCrosstalkSources_1031704_19.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash a4e0cf1b8b4cef2f7e7403b919362307 parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cc96b6fb-f930-4e6b-8c70-8f79c91354fd_isrForCrosstalkSources_1031716_52.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cc96b6fb-f930-4e6b-8c70-8f79c91354fd_isrForCrosstalkSources_1031716_52.stderr parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 6c48f2340bcaf23204fadad5a264284a parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d51c18e4-ccd3-42ae-b5a8-5e1ab60492e5_isrForCrosstalkSources_1031737_55.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d51c18e4-ccd3-42ae-b5a8-5e1ab60492e5_isrForCrosstalkSources_1031737_55.stderr parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash 7415217b884ad24cd70e774d08671594 parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2e9cc4bb-dcdd-443b-9dfd-c4b950d726aa_isrForCrosstalkSources_1031704_5.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2e9cc4bb-dcdd-443b-9dfd-c4b950d726aa_isrForCrosstalkSources_1031704_5.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash 6a8f839099940c24373db79a0cb473c3 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4bca09c3-292f-4b32-82a0-3226084b0eec_isrForCrosstalkSources_1031738_19.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4bca09c3-292f-4b32-82a0-3226084b0eec_isrForCrosstalkSources_1031738_19.stderr parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 5c63ad975960ede986aa698a7a818911 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/18d2a18b-30cf-42b3-b4b8-e438087af2f1_isrForCrosstalkSources_1031728_56.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/18d2a18b-30cf-42b3-b4b8-e438087af2f1_isrForCrosstalkSources_1031728_56.stderr parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 76203d915b933ff949e18e07b7fffe10 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/988ec25f-21a4-4780-a85f-31b144e36c6b_isrForCrosstalkSources_1031733_14.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/988ec25f-21a4-4780-a85f-31b144e36c6b_isrForCrosstalkSources_1031733_14.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 35ada41cef3a326ef313c2e939cc5446 parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c318b420-6301-4be4-a1c8-d35c3f104942_isrForCrosstalkSources_1031712_16.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c318b420-6301-4be4-a1c8-d35c3f104942_isrForCrosstalkSources_1031712_16.stderr parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash c549d8b3210819f90c9fc14168f1f0d1 parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/c80a7cc9-a783-4a01-9d4f-69488553d85a_isrForCrosstalkSources_1031701_8.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/c80a7cc9-a783-4a01-9d4f-69488553d85a_isrForCrosstalkSources_1031701_8.stderr parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash ed213c03abf51fca63585954d5d97de0 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/49a2996f-be64-45e9-8f09-e41001d79939_isrForCrosstalkSources_1031722_33.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/49a2996f-be64-45e9-8f09-e41001d79939_isrForCrosstalkSources_1031722_33.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash a79be44a3120690b7cbd1860680b2798 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f5cb15a0-99e3-4ce6-be47-447cadc94128_isrForCrosstalkSources_1031723_26.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f5cb15a0-99e3-4ce6-be47-447cadc94128_isrForCrosstalkSources_1031723_26.stderr parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 1d2dea2d726b5e1c5ac5382965b98991 parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/16f4879c-16b2-4d41-966f-273e34ab3e8c_isrForCrosstalkSources_1031703_40.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/16f4879c-16b2-4d41-966f-273e34ab3e8c_isrForCrosstalkSources_1031703_40.stderr parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 851005decd5384da57216630d4a883c2 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/0eac1ee7-ce95-4d56-8273-8895a9f49deb_isrForCrosstalkSources_1031709_52.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/0eac1ee7-ce95-4d56-8273-8895a9f49deb_isrForCrosstalkSources_1031709_52.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 806512699f284bd039bccabb5f40fb1a parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7248e68c-e933-403a-8de8-4e5195ebe6b2_isrForCrosstalkSources_1031732_41.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7248e68c-e933-403a-8de8-4e5195ebe6b2_isrForCrosstalkSources_1031732_41.stderr parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 5d3fe3353f7784158c274f79330c02d9 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/b025b3ae-8ba9-4d6e-b32f-1bc7aeed1d03_isrForCrosstalkSources_1031730_44.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/b025b3ae-8ba9-4d6e-b32f-1bc7aeed1d03_isrForCrosstalkSources_1031730_44.stderr parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 81a3a8d732e256998f06869fdebbbc49 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/5df39d20-69b1-4607-af02-7adb349ff84f_isrForCrosstalkSources_1031734_17.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/5df39d20-69b1-4607-af02-7adb349ff84f_isrForCrosstalkSources_1031734_17.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 2a17e684922df059319772f2e60e0aff parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/3cbfe7c4-c63f-449c-8df9-6a56314ac3b3_isrForCrosstalkSources_1031730_2.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/3cbfe7c4-c63f-449c-8df9-6a56314ac3b3_isrForCrosstalkSources_1031730_2.stderr parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash c629707624ba3e9090437a55f1fce2c1 parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/260547d1-38e6-4a94-b29f-c05b4a9d13f4_isrForCrosstalkSources_1031711_51.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/260547d1-38e6-4a94-b29f-c05b4a9d13f4_isrForCrosstalkSources_1031711_51.stderr parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 7fef62d5abd1229875d8300eb496d25e parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ab2ef105-5f7f-4233-ab67-d373077e56dc_isrForCrosstalkSources_1031715_26.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ab2ef105-5f7f-4233-ab67-d373077e56dc_isrForCrosstalkSources_1031715_26.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash ff3f0b0810115664b1156dca48aed021 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5d026d10-8623-4a23-8496-47bdfb573774_isrForCrosstalkSources_1031705_8.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5d026d10-8623-4a23-8496-47bdfb573774_isrForCrosstalkSources_1031705_8.stderr parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 1a1349c20375c0a24873d66c892e4fec parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/25feb15b-3fae-48d4-adbe-a86b1d3801fc_isrForCrosstalkSources_1031714_47.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/25feb15b-3fae-48d4-adbe-a86b1d3801fc_isrForCrosstalkSources_1031714_47.stderr parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 236fbe3e4a2d9fef664f535d339aa27c parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/4506989d-3590-4180-84cc-f1a28afe7c96_isrForCrosstalkSources_1031722_57.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/4506989d-3590-4180-84cc-f1a28afe7c96_isrForCrosstalkSources_1031722_57.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 3acb2f27f6c76501017f1c1682f80185 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/54578a5c-d2b1-4a52-97eb-d1efbe9b7602_isrForCrosstalkSources_1031713_34.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/54578a5c-d2b1-4a52-97eb-d1efbe9b7602_isrForCrosstalkSources_1031713_34.stderr parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 0784cd89c41b5ac94ebcddb04003bf06 parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/12f1faaf-098d-476c-a625-1129b0b1ed45_isrForCrosstalkSources_1031730_41.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/12f1faaf-098d-476c-a625-1129b0b1ed45_isrForCrosstalkSources_1031730_41.stderr parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash ed6c014e27901b3a830cb7f7f5257b86 parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8778c575-0da1-426b-8962-356d69165534_isrForCrosstalkSources_1031735_27.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8778c575-0da1-426b-8962-356d69165534_isrForCrosstalkSources_1031735_27.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 1798addc09879b429926baaebd9c6a6c parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6bb6ffe4-250b-4290-a588-d6ac6bdd7418_isrForCrosstalkSources_1031716_28.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6bb6ffe4-250b-4290-a588-d6ac6bdd7418_isrForCrosstalkSources_1031716_28.stderr parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash c90552476606c3fb4147dfbd9718f110 parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/22e50df8-8715-40e1-9a83-dfbb7f2101b3_isrForCrosstalkSources_1031735_33.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/22e50df8-8715-40e1-9a83-dfbb7f2101b3_isrForCrosstalkSources_1031735_33.stderr parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash 7f84047476676200ddebc2c2976750ab parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/255c6649-ebd6-4249-9707-122b3961f676_isrForCrosstalkSources_1031732_57.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/255c6649-ebd6-4249-9707-122b3961f676_isrForCrosstalkSources_1031732_57.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash b33514fde644c6459e49c68e2cd433ad parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/13fe53f9-b620-4dda-99ff-99e7559679f4_isrForCrosstalkSources_1031728_53.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/13fe53f9-b620-4dda-99ff-99e7559679f4_isrForCrosstalkSources_1031728_53.stderr parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 5726aef0eed1e30b31ac421c2a2b452c parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/4aab500c-3e8f-4e2f-9ce6-650cd5b5573e_isrForCrosstalkSources_1031725_36.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/4aab500c-3e8f-4e2f-9ce6-650cd5b5573e_isrForCrosstalkSources_1031725_36.stderr parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 4c9a7bc33d5cf214a6c41bc21d82b486 parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2fbde783-0e97-445a-b08c-ff74a01045ec_isrForCrosstalkSources_1031736_10.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2fbde783-0e97-445a-b08c-ff74a01045ec_isrForCrosstalkSources_1031736_10.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 4acdcb41ae8c4fb2e8d72ffcf7524fc3 parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/f2f9a5b4-c99b-477c-9271-ba7aa2764cac_isrForCrosstalkSources_1031711_25.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/f2f9a5b4-c99b-477c-9271-ba7aa2764cac_isrForCrosstalkSources_1031711_25.stderr parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash a382e226442a734488d7cc2ec524d39f parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/eac9e417-f421-4608-9d2d-faebf6cd9858_isrForCrosstalkSources_1031724_20.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/eac9e417-f421-4608-9d2d-faebf6cd9858_isrForCrosstalkSources_1031724_20.stderr parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash 9389003a9013c8f81efd13675ad2b15d parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/34ab5aec-4b0b-40fc-a0c0-3c259594e0e5_isrForCrosstalkSources_1031719_58.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/34ab5aec-4b0b-40fc-a0c0-3c259594e0e5_isrForCrosstalkSources_1031719_58.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 2407cbcc07f7b1d18cc1c5b953d0b99d parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c9a1320d-9f51-4e57-8dbd-2005398775ec_isrForCrosstalkSources_1031702_11.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c9a1320d-9f51-4e57-8dbd-2005398775ec_isrForCrosstalkSources_1031702_11.stderr parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 1ee798017e3983b16365ccec4b70a112 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8e75c1a5-85de-48fa-90b5-2b7c8224e0e3_isrForCrosstalkSources_1031735_50.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8e75c1a5-85de-48fa-90b5-2b7c8224e0e3_isrForCrosstalkSources_1031735_50.stderr parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 3f9a86662e3167baf44fda88b8982f7f parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e4dce8b9-b9c7-4fc4-afa5-c2947085ea81_isrForCrosstalkSources_1031716_39.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e4dce8b9-b9c7-4fc4-afa5-c2947085ea81_isrForCrosstalkSources_1031716_39.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 7594b55299bb836e4be2d71bea433f4c parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/49140586-e353-4e42-a77e-d54eba0436b9_isrForCrosstalkSources_1031709_41.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/49140586-e353-4e42-a77e-d54eba0436b9_isrForCrosstalkSources_1031709_41.stderr parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash c5a764ea79251d305e4da4dd250dd6c1 parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/dde79f33-6b40-49b0-9804-305ca8939b2a_isrForCrosstalkSources_1031701_35.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/dde79f33-6b40-49b0-9804-305ca8939b2a_isrForCrosstalkSources_1031701_35.stderr parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 475b7b4f383d26f68ca1d2ec6d5a37b8 parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/59990dfb-12fc-4ebe-8835-d3d9d5cdf2dd_isrForCrosstalkSources_1031702_53.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/59990dfb-12fc-4ebe-8835-d3d9d5cdf2dd_isrForCrosstalkSources_1031702_53.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash bb2f10a279e69df85958ef310904eb53 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c377a161-6f72-4309-b775-e6389c4e106c_isrForCrosstalkSources_1031703_53.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c377a161-6f72-4309-b775-e6389c4e106c_isrForCrosstalkSources_1031703_53.stderr parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash ff0969037b32046cef336b47858c4243 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/9b2da222-db3f-4cdd-92cb-55648dc421dd_isrForCrosstalkSources_1031720_16.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/9b2da222-db3f-4cdd-92cb-55648dc421dd_isrForCrosstalkSources_1031720_16.stderr parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 036371fa4e2a89f9cf971665390928b7 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4f8e1dd3-f12a-4869-b6d1-ec3806c9a590_isrForCrosstalkSources_1031732_14.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4f8e1dd3-f12a-4869-b6d1-ec3806c9a590_isrForCrosstalkSources_1031732_14.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 121c3dc2d094aa3d5708d5341b8590b0 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c831a806-5b9f-40e6-82c2-c45dcbde818d_isrForCrosstalkSources_1031738_55.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c831a806-5b9f-40e6-82c2-c45dcbde818d_isrForCrosstalkSources_1031738_55.stderr parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash de5fd35da03411b33edd0128ed7f2922 parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4127eaf3-fbca-4e2d-b9aa-fb747799377a_isrForCrosstalkSources_1031723_28.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4127eaf3-fbca-4e2d-b9aa-fb747799377a_isrForCrosstalkSources_1031723_28.stderr parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 4b4178f04137c3ac22cda8f72433a8d4 parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/5a00d4f5-1b33-400e-ac23-1536997e086e_isrForCrosstalkSources_1031737_43.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/5a00d4f5-1b33-400e-ac23-1536997e086e_isrForCrosstalkSources_1031737_43.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 45ccd9c87ff74cf19bff360548ed564e parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9c2f6284-5b13-4e00-906e-ec0a9fb658d1_isrForCrosstalkSources_1031733_11.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9c2f6284-5b13-4e00-906e-ec0a9fb658d1_isrForCrosstalkSources_1031733_11.stderr parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash fd3cfb17b115c2715d1a8daacb4791af parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/900ace74-a3e1-4a9a-882e-b3a846b83404_isrForCrosstalkSources_1031724_40.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/900ace74-a3e1-4a9a-882e-b3a846b83404_isrForCrosstalkSources_1031724_40.stderr parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 735444fbf35eadcefd16fa4e6973a952 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/2197f610-837c-4301-bd32-ac873eba30e7_isrForCrosstalkSources_1031699_44.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/2197f610-837c-4301-bd32-ac873eba30e7_isrForCrosstalkSources_1031699_44.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 10f65fd84706d5e5324dbe689164689a parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e8cced28-1905-4566-b185-be3f05185d90_isrForCrosstalkSources_1031728_3.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e8cced28-1905-4566-b185-be3f05185d90_isrForCrosstalkSources_1031728_3.stderr parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 9f666a0245cea32f189e3452c1470e18 parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/114caae0-cb3d-4203-bd24-6b41faaff8de_isrForCrosstalkSources_1031735_35.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/114caae0-cb3d-4203-bd24-6b41faaff8de_isrForCrosstalkSources_1031735_35.stderr parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 5cb675b5f2b682bb771df2da16d60230 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/81ea6377-b543-44a3-b943-58350c7063c5_isrForCrosstalkSources_1031715_56.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/81ea6377-b543-44a3-b943-58350c7063c5_isrForCrosstalkSources_1031715_56.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash d137c404f7fb18f0323b8fca6ed4aa65 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e78550d2-6541-4348-b6eb-d4932ee54e86_isrForCrosstalkSources_1031720_24.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e78550d2-6541-4348-b6eb-d4932ee54e86_isrForCrosstalkSources_1031720_24.stderr parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash c999b0fe92a671e32dbda2cc616ad3e5 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:05 proc_lsst.multi:170 [INFO] [multi] job_id 17147940 proc_lsst.multi INFO: [multi] job_id 17147940 2024-03-17 04:02:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 04:02:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147940 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147940 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17147940 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-17 04:02:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-17 04:02:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 04:02:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/fc211aa4-2cda-4a8d-8561-19a6134ed460_isrForCrosstalkSources_1031701_61.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/fc211aa4-2cda-4a8d-8561-19a6134ed460_isrForCrosstalkSources_1031701_61.stderr parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 6cc18ed6875f5b24531988380c65761a parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/01286372-aeb9-46ad-b153-f3052e7993a5_isrForCrosstalkSources_1031718_56.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/01286372-aeb9-46ad-b153-f3052e7993a5_isrForCrosstalkSources_1031718_56.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 1ac06d1500808e7c1b4a3e9641b20804 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/812de840-2341-49e9-ab78-bfb2470a59ae_isrForCrosstalkSources_1031737_58.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/812de840-2341-49e9-ab78-bfb2470a59ae_isrForCrosstalkSources_1031737_58.stderr parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 39bef463347d5fe63ce414c6e91806fc parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3d415f8e-8809-40ca-99b1-f3b0c43482b7_isrForCrosstalkSources_1031716_8.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3d415f8e-8809-40ca-99b1-f3b0c43482b7_isrForCrosstalkSources_1031716_8.stderr parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash cf3d91928345e50dea688544370e7ba8 parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/36ba444f-4f6f-46f9-8f08-face41e73143_isrForCrosstalkSources_1031704_30.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/36ba444f-4f6f-46f9-8f08-face41e73143_isrForCrosstalkSources_1031704_30.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 39848a992cb9d36ac1c978ebb96213c5 parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/e2eea651-8d2b-49d5-85fe-dc4a6a963a42_isrForCrosstalkSources_1031737_54.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/e2eea651-8d2b-49d5-85fe-dc4a6a963a42_isrForCrosstalkSources_1031737_54.stderr parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash b5990d5a5264b42c824dbc1439e2cf8f parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0b6d726b-1726-4d48-97b2-7cccfc18bee1_isrForCrosstalkSources_1031714_41.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0b6d726b-1726-4d48-97b2-7cccfc18bee1_isrForCrosstalkSources_1031714_41.stderr parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash e12c2d8c37c9b59329f134d2fa922859 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/951732c3-d88e-4de6-88d0-36818568b9d3_isrForCrosstalkSources_1031715_38.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/951732c3-d88e-4de6-88d0-36818568b9d3_isrForCrosstalkSources_1031715_38.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 06a2e82afbbd0f4678837f4ba88c2563 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4a3029ae-1d01-4103-bfb4-c44e86eec0fd_isrForCrosstalkSources_1031718_35.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4a3029ae-1d01-4103-bfb4-c44e86eec0fd_isrForCrosstalkSources_1031718_35.stderr parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash 62c8bbe58a5c6e5ffc9340088c5bf0b7 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/19ab9b10-f6a3-403f-96c8-49094a23272f_isrForCrosstalkSources_1031705_19.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/19ab9b10-f6a3-403f-96c8-49094a23272f_isrForCrosstalkSources_1031705_19.stderr parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash 81f21160380c1bf57e90bdcaa8558d43 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d37e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/829eed61-9181-4f19-9ece-e3c38a80b04e_isrForCrosstalkSources_1031736_13.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/829eed61-9181-4f19-9ece-e3c38a80b04e_isrForCrosstalkSources_1031736_13.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 5adea3fc3e95f1fa00eb366e40ccbc98 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/fc6c6241-936f-4723-9981-1c90ae7fafe3_isrForCrosstalkSources_1031725_50.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/fc6c6241-936f-4723-9981-1c90ae7fafe3_isrForCrosstalkSources_1031725_50.stderr parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 404f79a94a07cb649c149d594baae04c parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/7cf51e6e-5c92-4aa5-a523-a1ec523f93f5_isrForCrosstalkSources_1031724_4.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/7cf51e6e-5c92-4aa5-a523-a1ec523f93f5_isrForCrosstalkSources_1031724_4.stderr parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 7c2fb487a46dc6fff6d551efa6bbd2ae parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/25f58b1f-aaad-4c30-b5f3-36c9b1b8a7b3_isrForCrosstalkSources_1031705_56.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/25f58b1f-aaad-4c30-b5f3-36c9b1b8a7b3_isrForCrosstalkSources_1031705_56.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 91e7a26a3c3b833321f16ebeea5811bb parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/6e21091e-ef8f-465d-8adf-fbca7035563d_isrForCrosstalkSources_1031714_46.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/6e21091e-ef8f-465d-8adf-fbca7035563d_isrForCrosstalkSources_1031714_46.stderr parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 9d1086196f5540b438231005306db59f parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/659fbf53-7aac-4687-a844-6ac2d48034f0_isrForCrosstalkSources_1031712_49.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/659fbf53-7aac-4687-a844-6ac2d48034f0_isrForCrosstalkSources_1031712_49.stderr parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash d8358943673a368e1bb2cd00ef371def parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/f7c6eb6d-9da5-4c83-a1ac-7015e03f64f2_isrForCrosstalkSources_1031728_37.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/f7c6eb6d-9da5-4c83-a1ac-7015e03f64f2_isrForCrosstalkSources_1031728_37.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash b2b374cb204cb45ad124a9a1c5d08f11 parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/729a844a-7d7f-4f05-95cb-51aeacf0910a_isrForCrosstalkSources_1031724_26.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/729a844a-7d7f-4f05-95cb-51aeacf0910a_isrForCrosstalkSources_1031724_26.stderr parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash 00007b751c1125a53f49a245afdc5b0d parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/35a4d7e3-b0f2-4979-8d35-969ee3295d80_isrForCrosstalkSources_1031713_24.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/35a4d7e3-b0f2-4979-8d35-969ee3295d80_isrForCrosstalkSources_1031713_24.stderr parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash edce8d0c4ac22fd93760c4bdfcdb856d parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/35d662cd-d72a-4689-8593-687bb001d504_isrForCrosstalkSources_1031708_43.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/35d662cd-d72a-4689-8593-687bb001d504_isrForCrosstalkSources_1031708_43.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash 96591160d953026645a0f8c5a33d0521 parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6e225857-957b-4133-9064-30858df27fa6_isrForCrosstalkSources_1031721_6.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6e225857-957b-4133-9064-30858df27fa6_isrForCrosstalkSources_1031721_6.stderr parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash cea3e50c086cfda8a29337f65e137903 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a2164ffd-efda-464d-8ba1-4ef0b286d6cb_isrForCrosstalkSources_1031711_7.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a2164ffd-efda-464d-8ba1-4ef0b286d6cb_isrForCrosstalkSources_1031711_7.stderr parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash ff6b9073af2ddf16ea3eaebc9caedf31 parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4c613071-5920-455a-8633-a33dd2e19e7a_isrForCrosstalkSources_1031738_47.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4c613071-5920-455a-8633-a33dd2e19e7a_isrForCrosstalkSources_1031738_47.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash a5510b5fa5d953252fda5bc8113175de parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cdd15980-4299-43c1-b0ae-d89e9476cb2a_isrForCrosstalkSources_1031727_41.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cdd15980-4299-43c1-b0ae-d89e9476cb2a_isrForCrosstalkSources_1031727_41.stderr parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash c70a1ff1fac66dcfb48395c2d5035871 parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4bfbf50c-3afe-4b75-a1c9-0d456c0ccdb0_isrForCrosstalkSources_1031738_45.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4bfbf50c-3afe-4b75-a1c9-0d456c0ccdb0_isrForCrosstalkSources_1031738_45.stderr parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash 193189ca7f6ac9a50145c5fe29f947d4 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/7a19b6af-0dc2-4413-b452-d1cc718be8b7_isrForCrosstalkSources_1031717_2.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/7a19b6af-0dc2-4413-b452-d1cc718be8b7_isrForCrosstalkSources_1031717_2.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 5b557c72b15e27b21c2074ba180b4775 parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/f94cf3f3-2da5-4243-8189-b108329bf71b_isrForCrosstalkSources_1031734_6.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/f94cf3f3-2da5-4243-8189-b108329bf71b_isrForCrosstalkSources_1031734_6.stderr parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash 1a5bf4a137042c2708bc8093a25f6aa6 parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/34bb3a6f-aafb-4301-aa34-1ea105d4315c_isrForCrosstalkSources_1031716_36.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/34bb3a6f-aafb-4301-aa34-1ea105d4315c_isrForCrosstalkSources_1031716_36.stderr parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash e529170658a473762b5408c8d5b97d3d parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/1893b7eb-f761-4c0c-988b-ac766c9cc90c_isrForCrosstalkSources_1031714_33.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/1893b7eb-f761-4c0c-988b-ac766c9cc90c_isrForCrosstalkSources_1031714_33.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash 5c2dbed7f369f6fca303a5559c162eb6 parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6e695977-b440-4056-8f3e-31022cb366a6_isrForCrosstalkSources_1031723_19.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6e695977-b440-4056-8f3e-31022cb366a6_isrForCrosstalkSources_1031723_19.stderr parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash fedd1996d33d0bc069ad822ccedd5c9a parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f6bfb9ac-862e-410f-9fed-7909a3ba40bf_isrForCrosstalkSources_1031727_29.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f6bfb9ac-862e-410f-9fed-7909a3ba40bf_isrForCrosstalkSources_1031727_29.stderr parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash be55aa6d2793393f0b7aca0c93b8dd56 parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3b0adab2-c3c0-4310-b5ab-f0b29d35b0e6_isrForCrosstalkSources_1031736_50.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3b0adab2-c3c0-4310-b5ab-f0b29d35b0e6_isrForCrosstalkSources_1031736_50.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash bf1b3ba2b49745542ffa00eb572505be parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a30dab02-0844-4fc3-b256-e89f84352b38_isrForCrosstalkSources_1031731_41.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a30dab02-0844-4fc3-b256-e89f84352b38_isrForCrosstalkSources_1031731_41.stderr parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 139f4c2bb5025e9488f7f95245659e0c parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/f65afd00-5355-404d-88d1-6c2cef3206dd_isrForCrosstalkSources_1031712_5.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/f65afd00-5355-404d-88d1-6c2cef3206dd_isrForCrosstalkSources_1031712_5.stderr parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash a1246965ee9f65692fcccab0581c4e75 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/226d99d8-7871-4f6c-8b6a-72e0e1ae0e20_isrForCrosstalkSources_1031706_27.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/226d99d8-7871-4f6c-8b6a-72e0e1ae0e20_isrForCrosstalkSources_1031706_27.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash ef66498112ac5d1dc6a05bcdbc9a3ca6 parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/32594bca-6008-4b94-af84-607929ceacdd_isrForCrosstalkSources_1031709_13.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/32594bca-6008-4b94-af84-607929ceacdd_isrForCrosstalkSources_1031709_13.stderr parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash 35e6e5bcc9b52d1664b0d1167725f543 parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/fd30ee5b-aa09-4aa5-8cc5-27631ea2b425_isrForCrosstalkSources_1031724_1.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/fd30ee5b-aa09-4aa5-8cc5-27631ea2b425_isrForCrosstalkSources_1031724_1.stderr parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash 1db19111a79e0c9aafdd10d8d0226888 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/44db1db4-b147-4e60-807f-67602d36767a_isrForCrosstalkSources_1031727_23.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/44db1db4-b147-4e60-807f-67602d36767a_isrForCrosstalkSources_1031727_23.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash ad9407f7d2d50af6ff179892a2d49b71 parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/119ca187-d759-41f0-8606-ebba9416fa0a_isrForCrosstalkSources_1031724_45.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/119ca187-d759-41f0-8606-ebba9416fa0a_isrForCrosstalkSources_1031724_45.stderr parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash 1ff1b0fce73b7c666ef54ece6c5f057b parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/fd6a19c7-6a73-40a4-b311-75ac36348439_isrForCrosstalkSources_1031730_56.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/fd6a19c7-6a73-40a4-b311-75ac36348439_isrForCrosstalkSources_1031730_56.stderr parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 26eef18fd0f98e2735a73ecba4997fec parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:06 proc_lsst.multi:170 [INFO] [multi] job_id 17147941 proc_lsst.multi INFO: [multi] job_id 17147941 2024-03-17 04:02:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 04:02:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147941 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147941 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17147941 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-17 04:02:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-17 04:02:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 04:02:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e91f28f3-8aa2-4519-a222-52f7639bc09a_isrForCrosstalkSources_1031706_44.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e91f28f3-8aa2-4519-a222-52f7639bc09a_isrForCrosstalkSources_1031706_44.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 4454160524e616e02b15fb989042cdba parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/2488e2f2-f445-43b1-b29f-8b78a3f86c0b_isrForCrosstalkSources_1031732_5.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/2488e2f2-f445-43b1-b29f-8b78a3f86c0b_isrForCrosstalkSources_1031732_5.stderr parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash fdf4db395045a0b5d86bbd36cefcdc41 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cdf9e3a4-b640-4407-8b90-9f26f848e934_isrForCrosstalkSources_1031727_39.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cdf9e3a4-b640-4407-8b90-9f26f848e934_isrForCrosstalkSources_1031727_39.stderr parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 57f53f6d488f701ce72f09ffab7e9bf2 parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/afa10d15-6fbd-4c03-8c37-e430c955295d_isrForCrosstalkSources_1031713_54.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/afa10d15-6fbd-4c03-8c37-e430c955295d_isrForCrosstalkSources_1031713_54.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 07f5ceb1d58c7cb1cc7dbec4e90abe6f parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/012b02fd-acdc-46b3-8b1d-8c4eda7a340d_isrForCrosstalkSources_1031733_7.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/012b02fd-acdc-46b3-8b1d-8c4eda7a340d_isrForCrosstalkSources_1031733_7.stderr parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash 47fd24d31c70d8788bdcde10e5644f35 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5c0ae407-cc64-4db4-8692-52eddac4fbf2_isrForCrosstalkSources_1031703_23.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5c0ae407-cc64-4db4-8692-52eddac4fbf2_isrForCrosstalkSources_1031703_23.stderr parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 6f6c59798e8413ec6b941d172b53c985 parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/db182699-a23c-4d41-a52f-f7a014053f6b_isrForCrosstalkSources_1031733_43.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/db182699-a23c-4d41-a52f-f7a014053f6b_isrForCrosstalkSources_1031733_43.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 6f5fba04361195e4513c3b6fd33290e2 parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/76ba6ff4-42bc-4d74-886e-30fa92b83df0_isrForCrosstalkSources_1031709_37.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/76ba6ff4-42bc-4d74-886e-30fa92b83df0_isrForCrosstalkSources_1031709_37.stderr parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash c446ad5f471d0e8a6750a1469b88e90e parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1e630d95-c07b-42a3-84f2-05b589305495_isrForCrosstalkSources_1031709_24.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1e630d95-c07b-42a3-84f2-05b589305495_isrForCrosstalkSources_1031709_24.stderr parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash bc98c5a355151b9ab125aeb98daf94cb parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd840e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/16a9ddf9-991f-4d14-986c-5bcc094fa110_isrForCrosstalkSources_1031712_11.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/16a9ddf9-991f-4d14-986c-5bcc094fa110_isrForCrosstalkSources_1031712_11.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 95b4fba1f4a1c2bca94c52d1264219f1 parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e0393250-be73-4894-827a-1f083db5e4df_isrForCrosstalkSources_1031710_36.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e0393250-be73-4894-827a-1f083db5e4df_isrForCrosstalkSources_1031710_36.stderr parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash a9c5c0910e5a09268ef511134b7e86dc parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7cad4730-bfcb-412f-8e20-ec0d376d0023_isrForCrosstalkSources_1031735_31.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7cad4730-bfcb-412f-8e20-ec0d376d0023_isrForCrosstalkSources_1031735_31.stderr parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash ecce16df8fd3dd69c2a4e094e25fba3e parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd845e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/cfb61a35-db48-4413-b806-6656e0e6e8f1_isrForCrosstalkSources_1031734_40.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/cfb61a35-db48-4413-b806-6656e0e6e8f1_isrForCrosstalkSources_1031734_40.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash ccce4e9604e982527404c17d17febbf4 parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd842c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c3945e4e-f3a0-4100-bacd-ef6dfb0f218d_isrForCrosstalkSources_1031722_43.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c3945e4e-f3a0-4100-bacd-ef6dfb0f218d_isrForCrosstalkSources_1031722_43.stderr parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash 8b2c8aebf6f59df30110eaf47c5ec975 parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/5619983a-d257-47d6-90b9-47ec204d0afa_isrForCrosstalkSources_1031738_6.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/5619983a-d257-47d6-90b9-47ec204d0afa_isrForCrosstalkSources_1031738_6.stderr parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash e095c991ebf2ca5fbf98eaf5b0fb8a74 parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/28313962-4c12-4ec7-a674-bf49be8cd37f_isrForCrosstalkSources_1031727_61.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/28313962-4c12-4ec7-a674-bf49be8cd37f_isrForCrosstalkSources_1031727_61.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash 9d541584212f5e181a8e306ff704cdbc parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd844a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/a309025b-da64-441e-a2be-2c00d04d28ab_isrForCrosstalkSources_1031732_17.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/a309025b-da64-441e-a2be-2c00d04d28ab_isrForCrosstalkSources_1031732_17.stderr parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash 74574dd2e2e4309311491d4c1f43c35d parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/810abadd-e6e2-465a-8ea8-c6509b1ab6d5_isrForCrosstalkSources_1031722_5.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/810abadd-e6e2-465a-8ea8-c6509b1ab6d5_isrForCrosstalkSources_1031722_5.stderr parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash 5d736337812431b488a2ace3d42e0782 parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd847c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e7f89c59-7bf3-48b8-9218-ce54249343cd_isrForCrosstalkSources_1031704_50.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e7f89c59-7bf3-48b8-9218-ce54249343cd_isrForCrosstalkSources_1031704_50.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 12d7d9511755e160efca6f2de9fe8410 parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3d7bcc40-86c5-40fe-9798-df9af75d01c7_isrForCrosstalkSources_1031732_39.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3d7bcc40-86c5-40fe-9798-df9af75d01c7_isrForCrosstalkSources_1031732_39.stderr parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash 3859079e1b8721c7ae11e96b9ea1f662 parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd849a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/9c7e9076-28b1-4e42-960a-c9eb3a483795_isrForCrosstalkSources_1031722_60.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/9c7e9076-28b1-4e42-960a-c9eb3a483795_isrForCrosstalkSources_1031722_60.stderr parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash a748cc07b88404148ba8e678a61d3b12 parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f2dbaa67-58da-4b7e-9e15-71ab12506da0_isrForCrosstalkSources_1031721_17.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f2dbaa67-58da-4b7e-9e15-71ab12506da0_isrForCrosstalkSources_1031721_17.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 328f83ce18e638be98ea0bc4d4776af4 parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5931ecac-dee2-4a94-84be-20582c4cd51b_isrForCrosstalkSources_1031701_37.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5931ecac-dee2-4a94-84be-20582c4cd51b_isrForCrosstalkSources_1031701_37.stderr parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash c2b38fa59c39278517c0f08ef0558a58 parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d73e8676-6af5-4ee7-8170-2927c3a3526a_isrForCrosstalkSources_1031728_1.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d73e8676-6af5-4ee7-8170-2927c3a3526a_isrForCrosstalkSources_1031728_1.stderr parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 2ac0406ac135652bd96dd3cbd8caf019 parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/6276af7a-37fc-4447-bbd5-80b558c26789_isrForCrosstalkSources_1031702_27.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/6276af7a-37fc-4447-bbd5-80b558c26789_isrForCrosstalkSources_1031702_27.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 13c1f0ad5e0c433b54a1e66cc8534eb9 parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/85fff9bb-eaa2-489a-9785-f46f8f6224ec_isrForCrosstalkSources_1031735_14.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/85fff9bb-eaa2-489a-9785-f46f8f6224ec_isrForCrosstalkSources_1031735_14.stderr parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash d70c940c36c97170c11980eae9954052 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/c8519919-774a-45b5-b93f-122021ce7c04_isrForCrosstalkSources_1031733_23.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/c8519919-774a-45b5-b93f-122021ce7c04_isrForCrosstalkSources_1031733_23.stderr parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 34e1037ce1d56e11c073123fd7d45c2c parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/49fc1b62-fada-4cdc-8f97-cfa66cb182f1_isrForCrosstalkSources_1031738_58.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/49fc1b62-fada-4cdc-8f97-cfa66cb182f1_isrForCrosstalkSources_1031738_58.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash 6f6bb61d95f0baf6bb080a0fd85d3dc2 parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/263cc0a7-a2eb-498f-b260-6b531aba9d70_isrForCrosstalkSources_1031729_40.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/263cc0a7-a2eb-498f-b260-6b531aba9d70_isrForCrosstalkSources_1031729_40.stderr parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 329509e7bfdf45c3309f91a4cc1e56f7 parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4cfda397-da65-40e9-94b3-7f21fa19e298_isrForCrosstalkSources_1031709_29.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4cfda397-da65-40e9-94b3-7f21fa19e298_isrForCrosstalkSources_1031709_29.stderr parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash c8a2d7d1ae75d1e00729650218ae353a parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd859e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f38b6f34-b696-46ed-8977-de68c84cd252_isrForCrosstalkSources_1031699_61.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f38b6f34-b696-46ed-8977-de68c84cd252_isrForCrosstalkSources_1031699_61.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash ada2d7c57898aed4d421d92192e4065b parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd858a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c84d6e2e-0e29-4bba-994d-a6b1ac088bf7_isrForCrosstalkSources_1031707_46.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c84d6e2e-0e29-4bba-994d-a6b1ac088bf7_isrForCrosstalkSources_1031707_46.stderr parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash 091d401f3e9c96b64780a2f23f156ede parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/281db6ac-2ceb-4f74-8caf-2378f2356e4f_isrForCrosstalkSources_1031703_10.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/281db6ac-2ceb-4f74-8caf-2378f2356e4f_isrForCrosstalkSources_1031703_10.stderr parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash 92fdc5b93376b3a38d72cd840b2babc9 parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/70d0c683-0dbc-4ce6-a5b4-a1dedc2ea2ae_isrForCrosstalkSources_1031717_49.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/70d0c683-0dbc-4ce6-a5b4-a1dedc2ea2ae_isrForCrosstalkSources_1031717_49.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 31f622659e99a241554f4333987247cb parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/35f62a9e-3df8-455a-98e0-d4bd39c71f1d_isrForCrosstalkSources_1031699_59.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/35f62a9e-3df8-455a-98e0-d4bd39c71f1d_isrForCrosstalkSources_1031699_59.stderr parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 9590846cc9a1469760938faf74ae8a11 parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd856c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/de672874-a202-4a79-816d-127650ab3600_isrForCrosstalkSources_1031724_29.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/de672874-a202-4a79-816d-127650ab3600_isrForCrosstalkSources_1031724_29.stderr parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 650e7c710fe4b9c867dbdbc2440ae3f5 parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/74369388-da3e-4f34-9f65-41852c4df044_isrForCrosstalkSources_1031729_27.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/74369388-da3e-4f34-9f65-41852c4df044_isrForCrosstalkSources_1031729_27.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash d6e3f574e39aeef58a0e037ef7bfd4f1 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/73323920-e1bc-4d3d-9b07-8439cb40dfff_isrForCrosstalkSources_1031708_41.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/73323920-e1bc-4d3d-9b07-8439cb40dfff_isrForCrosstalkSources_1031708_41.stderr parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 9dc9cc9d9ea9d5bd4cccee58e6f15995 parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0556f597-0192-4984-bef6-4e26c8c6fde3_isrForCrosstalkSources_1031724_27.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0556f597-0192-4984-bef6-4e26c8c6fde3_isrForCrosstalkSources_1031724_27.stderr parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 7271d2bdaeec185bba9c16abaa10b249 parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/03222f5e-c105-416d-a90a-1aae40913e9d_isrForCrosstalkSources_1031736_4.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/03222f5e-c105-416d-a90a-1aae40913e9d_isrForCrosstalkSources_1031736_4.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 145d7a33ee49f0a3e21c28d688c2b008 parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/34eac1bb-0834-442b-ab32-a105054dc8a8_isrForCrosstalkSources_1031720_34.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/34eac1bb-0834-442b-ab32-a105054dc8a8_isrForCrosstalkSources_1031720_34.stderr parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash fcd92b2624cb9d932507ebd4e436f091 parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/496af609-ad3b-41cb-a214-a1a15d59fba4_isrForCrosstalkSources_1031704_60.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/496af609-ad3b-41cb-a214-a1a15d59fba4_isrForCrosstalkSources_1031704_60.stderr parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash e6a8a7145363dacb8620d53e8e2c3b6c parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/18c0f02f-8a7c-4763-9310-930153ed2ef4_isrForCrosstalkSources_1031703_38.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/18c0f02f-8a7c-4763-9310-930153ed2ef4_isrForCrosstalkSources_1031703_38.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash cb56df261b549c88971e7a56e8694732 parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd853a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c4a89fc1-0d79-410e-9125-fb8bc32cc172_isrForCrosstalkSources_1031738_35.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c4a89fc1-0d79-410e-9125-fb8bc32cc172_isrForCrosstalkSources_1031738_35.stderr parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash ed95193e2479890e2d007eec692859bf parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd85080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e23eedeb-a499-42a3-bb85-1a683ed314c0_isrForCrosstalkSources_1031717_12.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e23eedeb-a499-42a3-bb85-1a683ed314c0_isrForCrosstalkSources_1031717_12.stderr parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash a48a036b189b64da95572fec3eaed7b2 parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ccf95c83-bdb6-452a-b241-d1ad5cf9c095_isrForCrosstalkSources_1031711_9.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ccf95c83-bdb6-452a-b241-d1ad5cf9c095_isrForCrosstalkSources_1031711_9.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash 8def4b294bae98cb19f3ccd3b460788d parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/541d2996-81ad-4280-9ed4-f17a567f7606_isrForCrosstalkSources_1031712_13.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/541d2996-81ad-4280-9ed4-f17a567f7606_isrForCrosstalkSources_1031712_13.stderr parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash 88760060029385239f83b5b7f2dba4bb parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:06 proc_lsst.multi:170 [INFO] [multi] job_id 17147942 proc_lsst.multi INFO: [multi] job_id 17147942 2024-03-17 04:02:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-17 04:02:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147942 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147942 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17147942 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-17 04:02:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-17 04:02:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-17 04:02:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/107937c3-eb18-46d7-94a6-28c1f9d3d364_isrForCrosstalkSources_1031715_5.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/107937c3-eb18-46d7-94a6-28c1f9d3d364_isrForCrosstalkSources_1031715_5.stderr parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash d993b97b82ebcd4a659dfe69ca07cae5 parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/16309122-b3e4-4f87-8cff-a980f4dfcde4_isrForCrosstalkSources_1031735_16.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/16309122-b3e4-4f87-8cff-a980f4dfcde4_isrForCrosstalkSources_1031735_16.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash 88aec37ff5e8328756dc5d7b36879b46 parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/12178eb2-e8fe-4672-97a3-b4e909793693_isrForCrosstalkSources_1031720_36.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/12178eb2-e8fe-4672-97a3-b4e909793693_isrForCrosstalkSources_1031720_36.stderr parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 3361bfbf07e14109c708cb66e662766c parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5743b638-f37f-4cba-8afe-88eab5be4443_isrForCrosstalkSources_1031714_37.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5743b638-f37f-4cba-8afe-88eab5be4443_isrForCrosstalkSources_1031714_37.stderr parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash b4266d403c54938e05e79827d730d23c parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/71ac5674-01f4-4579-90bc-3bbd4fd13e47_isrForCrosstalkSources_1031705_51.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/71ac5674-01f4-4579-90bc-3bbd4fd13e47_isrForCrosstalkSources_1031705_51.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash bd054e2e33659da454b00469150d953e parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/3944cb57-6744-48fe-be5d-59feaca4e22d_isrForCrosstalkSources_1031715_34.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/3944cb57-6744-48fe-be5d-59feaca4e22d_isrForCrosstalkSources_1031715_34.stderr parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash 74193ab8435108e01088a43b347d4c21 parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/2cae5bd9-2ef2-4859-b685-f9ee51f9275e_isrForCrosstalkSources_1031721_9.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/2cae5bd9-2ef2-4859-b685-f9ee51f9275e_isrForCrosstalkSources_1031721_9.stderr parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash 672f5337ed6636ed1ebcfada14a3cfac parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd2d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e8fc6a15-a627-4f66-90a0-c343a4e6eecd_isrForCrosstalkSources_1031720_11.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e8fc6a15-a627-4f66-90a0-c343a4e6eecd_isrForCrosstalkSources_1031720_11.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash a7f0a6ab536474f13b533d2ae7b5c7fd parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/87daf0cc-dd80-4d2d-aef1-6ae47d2080b0_isrForCrosstalkSources_1031706_6.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/87daf0cc-dd80-4d2d-aef1-6ae47d2080b0_isrForCrosstalkSources_1031706_6.stderr parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 186a9557b9403427e151b6504ba22a47 parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/71e5985d-e887-4d4a-bf56-6f9ad091ffcd_isrForCrosstalkSources_1031722_36.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/71e5985d-e887-4d4a-bf56-6f9ad091ffcd_isrForCrosstalkSources_1031722_36.stderr parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash afb2fe2bb4ee897b6d1cc4fd6f15b83d parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/bf8ded62-c9e9-452e-9b4a-9189e2f3809d_isrForCrosstalkSources_1031708_9.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/bf8ded62-c9e9-452e-9b4a-9189e2f3809d_isrForCrosstalkSources_1031708_9.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 0a65cdf567ee04085a9bef97ea1a81e1 parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd871a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/1cbc9d8d-ff96-4344-95b7-3f834e08b017_isrForCrosstalkSources_1031702_41.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/1cbc9d8d-ff96-4344-95b7-3f834e08b017_isrForCrosstalkSources_1031702_41.stderr parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash 0f0203ef2bc7430bf5919e39041d6fb3 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0faebf59-59ff-4936-b1b2-d4f5007a208a_isrForCrosstalkSources_1031707_14.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0faebf59-59ff-4936-b1b2-d4f5007a208a_isrForCrosstalkSources_1031707_14.stderr parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash c7f869c23fab527bd4e360aa01cd22bb parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/436bacfe-65ae-427e-9d96-8c7f55153642_isrForCrosstalkSources_1031705_61.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/436bacfe-65ae-427e-9d96-8c7f55153642_isrForCrosstalkSources_1031705_61.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash d6f900cfdfbf307410d4b02661ad8b42 parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b37cf49e-364f-47bd-a73d-73d36f776c35_isrForCrosstalkSources_1031721_32.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b37cf49e-364f-47bd-a73d-73d36f776c35_isrForCrosstalkSources_1031721_32.stderr parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 74ab5fcb63a0e58ae44aaa31262f8cb9 parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3cd4d129-e1ca-4847-8d9b-4f37cd5ec806_isrForCrosstalkSources_1031723_11.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3cd4d129-e1ca-4847-8d9b-4f37cd5ec806_isrForCrosstalkSources_1031723_11.stderr parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash 297173ef0eb1b61612191a6b1197b62d parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4d0c6663-19e1-4184-9cc1-11cafec47e1b_isrForCrosstalkSources_1031723_23.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4d0c6663-19e1-4184-9cc1-11cafec47e1b_isrForCrosstalkSources_1031723_23.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash 94d4da9f98f2340aa251ddaa533ce47d parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d66c1a95-1e2d-48bd-a20a-2a3fba63a439_isrForCrosstalkSources_1031734_19.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d66c1a95-1e2d-48bd-a20a-2a3fba63a439_isrForCrosstalkSources_1031734_19.stderr parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 48228c66b02f9280b3356c491cc53d02 parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5febd1be-d407-40a0-948d-d0b8a8242543_isrForCrosstalkSources_1031730_1.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5febd1be-d407-40a0-948d-d0b8a8242543_isrForCrosstalkSources_1031730_1.stderr parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash 02b535fd4e4d91c8c3175d9f12b09f32 parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f7a211e1-f383-4386-a182-d56623dfedf4_isrForCrosstalkSources_1031700_43.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f7a211e1-f383-4386-a182-d56623dfedf4_isrForCrosstalkSources_1031700_43.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 90ca7276b5f4f0a8fd756a9f350f1c3e parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/270b19cf-c87e-4c92-9dee-e1c2cf8325c9_isrForCrosstalkSources_1031725_49.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/270b19cf-c87e-4c92-9dee-e1c2cf8325c9_isrForCrosstalkSources_1031725_49.stderr parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash 66e017fa564586d256df49c2e0474e4f parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/9f29e782-4085-4f24-9d41-28a65c3b311d_isrForCrosstalkSources_1031703_59.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/9f29e782-4085-4f24-9d41-28a65c3b311d_isrForCrosstalkSources_1031703_59.stderr parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash ca8d70d9a241980215e3e030af19baac parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/58ac916b-2d28-49d9-ae9f-fe1c19b69b89_isrForCrosstalkSources_1031701_46.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/58ac916b-2d28-49d9-ae9f-fe1c19b69b89_isrForCrosstalkSources_1031701_46.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 6b70a279e3abef1c38dccebeb13d5325 parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/870fe78e-89e8-4b94-8598-94325cf78da8_isrForCrosstalkSources_1031710_33.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/870fe78e-89e8-4b94-8598-94325cf78da8_isrForCrosstalkSources_1031710_33.stderr parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash 536b6d0529c86157b14bd24a5a1559a1 parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/64bcec3a-a7b3-46a6-8c5a-f74b7202b522_isrForCrosstalkSources_1031713_27.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/64bcec3a-a7b3-46a6-8c5a-f74b7202b522_isrForCrosstalkSources_1031713_27.stderr parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash a641852a8c4df0a2c99f0270b4fd260f parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/cde3304f-4065-4bf5-9d98-b8907e394a90_isrForCrosstalkSources_1031713_58.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/cde3304f-4065-4bf5-9d98-b8907e394a90_isrForCrosstalkSources_1031713_58.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 5dc27998e38a4418ef33cc663aa487f2 parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd867a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/1fe9b2b4-cfcf-4219-91f8-70a95276b99e_isrForCrosstalkSources_1031708_25.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/1fe9b2b4-cfcf-4219-91f8-70a95276b99e_isrForCrosstalkSources_1031708_25.stderr parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash be9615e9377b95baaa79b90563f92fe3 parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2f0b08dc-15c4-498c-a0f7-30ac4e781e08_isrForCrosstalkSources_1031709_54.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2f0b08dc-15c4-498c-a0f7-30ac4e781e08_isrForCrosstalkSources_1031709_54.stderr parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash 0079101c1af1b2f932ce64d5ee5231e9 parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/10ae0c34-60a6-42f6-9ae8-5707a25ff2c4_isrForCrosstalkSources_1031711_55.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/10ae0c34-60a6-42f6-9ae8-5707a25ff2c4_isrForCrosstalkSources_1031711_55.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash 2e9220cbab89ef0e6674fc12f465d1b0 parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd868e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/cd9e0c78-5a95-44bc-b1c2-37d76d8ee46c_isrForCrosstalkSources_1031722_11.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/cd9e0c78-5a95-44bc-b1c2-37d76d8ee46c_isrForCrosstalkSources_1031722_11.stderr parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash 0a32bb1feaac4befa45b36b3c6d12b57 parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/f85df925-74ff-431f-a01b-21f591319150_isrForCrosstalkSources_1031738_29.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/f85df925-74ff-431f-a01b-21f591319150_isrForCrosstalkSources_1031738_29.stderr parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash d82526fa36e9f26237f708f5caba8942 parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2681fa5c-14a0-4ab6-91e0-9ecd3ce2d4d3_isrForCrosstalkSources_1031724_14.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2681fa5c-14a0-4ab6-91e0-9ecd3ce2d4d3_isrForCrosstalkSources_1031724_14.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash 4023d0f9b90e9d0528974032084293db parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/5dd52ba8-97e0-4bc1-9a7e-621bbfa00c6f_isrForCrosstalkSources_1031734_50.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/5dd52ba8-97e0-4bc1-9a7e-621bbfa00c6f_isrForCrosstalkSources_1031734_50.stderr parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash 32bd33ac73f078e3041ec3342714803b parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd863e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/fad0b31a-50b3-4634-9616-1ee9819158b1_isrForCrosstalkSources_1031727_18.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/fad0b31a-50b3-4634-9616-1ee9819158b1_isrForCrosstalkSources_1031727_18.stderr parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 16a1a1348062442562be35ab8e6e1481 parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd865c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/bb37b8b1-a71d-4e3c-8586-99a9a6e7d885_isrForCrosstalkSources_1031712_1.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/bb37b8b1-a71d-4e3c-8586-99a9a6e7d885_isrForCrosstalkSources_1031712_1.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash c2fc3194c1c34f413f856414c9cf7426 parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd862a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/74233290-0a52-4111-9c20-5b532134c1ff_isrForCrosstalkSources_1031720_5.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/74233290-0a52-4111-9c20-5b532134c1ff_isrForCrosstalkSources_1031720_5.stderr parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash b87541457daee73f0543f0c3a0719d15 parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b717c230-4150-4263-aa75-b3ff00f1f75e_isrForCrosstalkSources_1031707_11.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b717c230-4150-4263-aa75-b3ff00f1f75e_isrForCrosstalkSources_1031707_11.stderr parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 4a69952cc65eab21e0b399ae30f6001e parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2b5e6f66-e0e0-46c4-a165-b417a3c9547c_isrForCrosstalkSources_1031704_3.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2b5e6f66-e0e0-46c4-a165-b417a3c9547c_isrForCrosstalkSources_1031704_3.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash 8912053711838f6b9e4ec38d21278c28 parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd860c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/4e97ebd5-0065-4c88-9e74-940a0dd02655_isrForCrosstalkSources_1031729_24.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/4e97ebd5-0065-4c88-9e74-940a0dd02655_isrForCrosstalkSources_1031729_24.stderr parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 63788038a1a7ef3fe1c765d6bf2ef722 parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/c2e8043b-ce64-401e-813e-2295ad07e98e_isrForCrosstalkSources_1031724_6.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/c2e8043b-ce64-401e-813e-2295ad07e98e_isrForCrosstalkSources_1031724_6.stderr parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash 3446a14b964f64ba81819481fb6658a8 parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd874c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3f7aaf5a-9e60-44bc-affe-edf01f2024c3_isrForCrosstalkSources_1031732_7.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3f7aaf5a-9e60-44bc-affe-edf01f2024c3_isrForCrosstalkSources_1031732_7.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 48827473ca2096d54feac76fc1641ab4 parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/71b354bf-0968-4dca-b3a3-79ef80d19c4a_isrForCrosstalkSources_1031709_55.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/71b354bf-0968-4dca-b3a3-79ef80d19c4a_isrForCrosstalkSources_1031709_55.stderr parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash aab672f3d55ffb18cccd39397901ea9b parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e4efd48c-997d-4a86-8f1a-20e3dba11650_isrForCrosstalkSources_1031717_30.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e4efd48c-997d-4a86-8f1a-20e3dba11650_isrForCrosstalkSources_1031717_30.stderr parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash cef07808c85337565235b2e482839033 parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd876a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/185a99cd-630e-4b30-8a7f-f559a08db134_isrForCrosstalkSources_1031723_13.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/185a99cd-630e-4b30-8a7f-f559a08db134_isrForCrosstalkSources_1031723_13.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 889ee4843348b7d2105930c4d44bb25c parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c88d44e9-8f0f-4526-a149-e90d670c8162_isrForCrosstalkSources_1031715_45.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c88d44e9-8f0f-4526-a149-e90d670c8162_isrForCrosstalkSources_1031715_45.stderr parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash c2e89323eee2e449633a4e49a95a507b parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd877e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/1f452135-cf6f-4806-b980-ddc87d7fca8f_isrForCrosstalkSources_1031727_6.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/1f452135-cf6f-4806-b980-ddc87d7fca8f_isrForCrosstalkSources_1031727_6.stderr parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 847fa2ccf0c68977216c89d54ac48b66 parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c94b1f63-01eb-463b-9475-4a763079f8be_isrForCrosstalkSources_1031725_28.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c94b1f63-01eb-463b-9475-4a763079f8be_isrForCrosstalkSources_1031725_28.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 5150d04b5d77082084649cf7333c06da parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e072f816-d490-45f3-a6d0-17516b027af7_isrForCrosstalkSources_1031738_28.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e072f816-d490-45f3-a6d0-17516b027af7_isrForCrosstalkSources_1031738_28.stderr parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash 0c5a54f8d127a41ee2ac66e68cc9b32e parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd879c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/07d7b59a-bb37-4b33-900b-50479d45b131_isrForCrosstalkSources_1031731_26.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/07d7b59a-bb37-4b33-900b-50479d45b131_isrForCrosstalkSources_1031731_26.stderr parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash 79b02de3483eb06089e899d1e0d05e9a parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/24054c14-51e0-4520-b74f-d3d3d0c9c1c9_isrForCrosstalkSources_1031722_9.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/24054c14-51e0-4520-b74f-d3d3d0c9c1c9_isrForCrosstalkSources_1031722_9.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash 8f0b76919fa306f9057eb21c8d033305 parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c480af5b-a820-4e7d-928c-9555b71ab6c4_isrForCrosstalkSources_1031706_42.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c480af5b-a820-4e7d-928c-9555b71ab6c4_isrForCrosstalkSources_1031706_42.stderr parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash 35d3197c08afd90d826f0f57c5365efa parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4d40d505-a680-4344-955f-25cb8c1c90ad_isrForCrosstalkSources_1031720_42.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4d40d505-a680-4344-955f-25cb8c1c90ad_isrForCrosstalkSources_1031720_42.stderr parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 25c357e705d452a2b06161126f9389a3 parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/50cae9c3-d7bb-4e2b-8e8c-6879f108397b_isrForCrosstalkSources_1031732_31.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/50cae9c3-d7bb-4e2b-8e8c-6879f108397b_isrForCrosstalkSources_1031732_31.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 77ccb132cb2c4dde50080af2e8e00766 parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/30bd2e2d-f62f-4904-8044-c81bfa8902bc_isrForCrosstalkSources_1031719_17.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/30bd2e2d-f62f-4904-8044-c81bfa8902bc_isrForCrosstalkSources_1031719_17.stderr parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash 4b471ac5ef765c51c4e0206c5a45c5ca parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/7cba93b2-6e13-4728-8a16-5eab091cb189_isrForCrosstalkSources_1031728_44.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/7cba93b2-6e13-4728-8a16-5eab091cb189_isrForCrosstalkSources_1031728_44.stderr parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 9ce6e974e257143aa86007a8777c6250 parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/53a2a7f4-79f5-4f17-8be6-1845f6a93221_isrForCrosstalkSources_1031709_49.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/53a2a7f4-79f5-4f17-8be6-1845f6a93221_isrForCrosstalkSources_1031709_49.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash fa928f6da027b9719ab589713c5ab5d8 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1fd2e7af-6228-4316-85bd-641d0ad2b23c_isrForCrosstalkSources_1031711_11.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1fd2e7af-6228-4316-85bd-641d0ad2b23c_isrForCrosstalkSources_1031711_11.stderr parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash fd3d78d016afda8b80b0b52b5fffb1d8 parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/da54ca92-c448-405e-b27e-f8035f7107cc_isrForCrosstalkSources_1031728_14.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/da54ca92-c448-405e-b27e-f8035f7107cc_isrForCrosstalkSources_1031728_14.stderr parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 50e59713264774282a2a4750394b0b4f parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/aa75606b-339d-41e6-aac8-bfd1d95fdb99_isrForCrosstalkSources_1031707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/aa75606b-339d-41e6-aac8-bfd1d95fdb99_isrForCrosstalkSources_1031707_31.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash 7f677d7599e6e43d991bff4f2cea55ce parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/7fe3d535-9a52-4813-b806-20ae06bc8a40_isrForCrosstalkSources_1031720_8.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/7fe3d535-9a52-4813-b806-20ae06bc8a40_isrForCrosstalkSources_1031720_8.stderr parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 9fc8b3f2537ab96417c5a98412535984 parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/977b737c-4008-4d47-8ed6-a92685f4609c_isrForCrosstalkSources_1031721_40.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/977b737c-4008-4d47-8ed6-a92685f4609c_isrForCrosstalkSources_1031721_40.stderr parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 26b889119d8e7a88a3418aacea493b73 parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8e52f881-bd8b-46d9-a939-f8ece7dad40f_isrForCrosstalkSources_1031719_21.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8e52f881-bd8b-46d9-a939-f8ece7dad40f_isrForCrosstalkSources_1031719_21.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash 8e1a567ffa74cceb37007ec5d8ea9339 parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/abe80ace-181b-405e-a2fd-e057ee62a7df_isrForCrosstalkSources_1031726_1.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/abe80ace-181b-405e-a2fd-e057ee62a7df_isrForCrosstalkSources_1031726_1.stderr parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 8071582ff651bae6aff56a082dba5af8 parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/38bc3794-d8c0-4744-a59c-c72fd5a530ce_isrForCrosstalkSources_1031723_50.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/38bc3794-d8c0-4744-a59c-c72fd5a530ce_isrForCrosstalkSources_1031723_50.stderr parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash 799d925a1f72fc2681200fd5ba07f8a5 parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/c81dd6a0-fa25-4a91-b192-fc80219b9640_isrForCrosstalkSources_1031720_39.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/c81dd6a0-fa25-4a91-b192-fc80219b9640_isrForCrosstalkSources_1031720_39.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 4eb387e779161422499564d704e10d7c parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/da416623-4ab6-4928-9a1d-4deaebdf8189_isrForCrosstalkSources_1031725_15.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/da416623-4ab6-4928-9a1d-4deaebdf8189_isrForCrosstalkSources_1031725_15.stderr parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 156aaebbe58d161ab4e1193514c374b5 parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/32861743-6122-4355-966e-8b0d4e33904f_isrForCrosstalkSources_1031705_49.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/32861743-6122-4355-966e-8b0d4e33904f_isrForCrosstalkSources_1031705_49.stderr parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash e4e75801575139cadb8bea3cd59ca8dc parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdccfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6c4344cd-282d-4909-87de-f9b3050fa2a1_isrForCrosstalkSources_1031721_53.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6c4344cd-282d-4909-87de-f9b3050fa2a1_isrForCrosstalkSources_1031721_53.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash 51d63265211c5ef91788b0c26f939c38 parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdccf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7cb3d3f7-609b-45f5-89e2-202d19be5b42_isrForCrosstalkSources_1031704_36.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7cb3d3f7-609b-45f5-89e2-202d19be5b42_isrForCrosstalkSources_1031704_36.stderr parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 49485a04643fb5560ec85df63992698c parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdccea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0d8074f3-4ee3-4052-90ab-b6bff094e87e_isrForCrosstalkSources_1031717_51.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0d8074f3-4ee3-4052-90ab-b6bff094e87e_isrForCrosstalkSources_1031717_51.stderr parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 22b03b6f57cb256f42b358ab9cc7108c parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdccd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/89b00491-c389-402c-a3e8-5f025b192999_isrForCrosstalkSources_1031712_37.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/89b00491-c389-402c-a3e8-5f025b192999_isrForCrosstalkSources_1031712_37.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash e093a0b14390742428015e1a492df85f parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/333f367c-94cf-410f-abf0-bd2f2878c297_isrForCrosstalkSources_1031718_42.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/333f367c-94cf-410f-abf0-bd2f2878c297_isrForCrosstalkSources_1031718_42.stderr parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash 9061a9f28a858d8e6790103b0b604b7e parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/1cc48962-9f3b-4cf9-b75a-ace6efb56350_isrForCrosstalkSources_1031722_14.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/1cc48962-9f3b-4cf9-b75a-ace6efb56350_isrForCrosstalkSources_1031722_14.stderr parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash babe61840006c3e2983eb0e0502b1c80 parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdccc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d8cca335-01a4-4d31-8421-867125c58163_isrForCrosstalkSources_1031728_21.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d8cca335-01a4-4d31-8421-867125c58163_isrForCrosstalkSources_1031728_21.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash d9e2afd23bf83c3f27db4cd6b12d372f parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdccb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d647867d-1c44-4715-bc6d-ed5d405e1242_isrForCrosstalkSources_1031727_43.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d647867d-1c44-4715-bc6d-ed5d405e1242_isrForCrosstalkSources_1031727_43.stderr parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 91ea41028dea304a79dee393ece23d65 parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ea614c9a-2363-4784-929d-d2a18c8d2ec6_isrForCrosstalkSources_1031709_26.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ea614c9a-2363-4784-929d-d2a18c8d2ec6_isrForCrosstalkSources_1031709_26.stderr parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash c04844a2a7a1f8a470e892261c88839d parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdccae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/2686c88b-7c23-410f-989e-292515fc22a2_isrForCrosstalkSources_1031718_3.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/2686c88b-7c23-410f-989e-292515fc22a2_isrForCrosstalkSources_1031718_3.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 08c8d9d7987d79209b7894e6391058a6 parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/54443b03-9fb4-4db6-927a-bcc84c685d64_isrForCrosstalkSources_1031727_35.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/54443b03-9fb4-4db6-927a-bcc84c685d64_isrForCrosstalkSources_1031727_35.stderr parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash 9fb61ebe3add6f2dc7f9a377cd974daa parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3964ac24-3d2c-42ff-a417-e705f39c3324_isrForCrosstalkSources_1031738_51.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3964ac24-3d2c-42ff-a417-e705f39c3324_isrForCrosstalkSources_1031738_51.stderr parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash d5b464d08ea2eb150e85c74b82775e45 parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7ad7794c-d3d2-4798-9487-710c9af00e4d_isrForCrosstalkSources_1031732_60.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7ad7794c-d3d2-4798-9487-710c9af00e4d_isrForCrosstalkSources_1031732_60.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash 2d064219fd40a6259510b60b874cc93d parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8c82ad7e-48bd-4cdc-af90-6a76b16ce429_isrForCrosstalkSources_1031736_17.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8c82ad7e-48bd-4cdc-af90-6a76b16ce429_isrForCrosstalkSources_1031736_17.stderr parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash 27411daa1007068257d466eff2897313 parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8a9be04f-f5b8-4f67-bd3b-d92a67326fd1_isrForCrosstalkSources_1031735_56.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8a9be04f-f5b8-4f67-bd3b-d92a67326fd1_isrForCrosstalkSources_1031735_56.stderr parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash ce852a61204134eb943061bced57a61d parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a2ef7ec3-ff79-48e1-ba7d-c11d21e82401_isrForCrosstalkSources_1031736_9.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a2ef7ec3-ff79-48e1-ba7d-c11d21e82401_isrForCrosstalkSources_1031736_9.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash c86a7ab17d2e738b7387236df57ba7fe parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f092f259-dbcb-4c4b-b4ca-0a083394e8a0_isrForCrosstalkSources_1031705_57.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f092f259-dbcb-4c4b-b4ca-0a083394e8a0_isrForCrosstalkSources_1031705_57.stderr parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash 9771ba77408fae031f422883d00f5f4f parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/b51b2cb0-a979-47cd-a782-7905b3c38ca3_isrForCrosstalkSources_1031727_31.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/b51b2cb0-a979-47cd-a782-7905b3c38ca3_isrForCrosstalkSources_1031727_31.stderr parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash 9058c291fee2a18e9a9bd65060bf8c1f parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e17821aa-8a47-4938-850b-b3b23a960142_isrForCrosstalkSources_1031709_5.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e17821aa-8a47-4938-850b-b3b23a960142_isrForCrosstalkSources_1031709_5.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 622015be225400e7d996fd8834301fe4 parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c954491b-e03f-48e9-a281-4ced61bfa3c4_isrForCrosstalkSources_1031729_39.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c954491b-e03f-48e9-a281-4ced61bfa3c4_isrForCrosstalkSources_1031729_39.stderr parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 2c9907c2bad3b00ca9d06042012df0e1 parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3659335a-5e14-4db9-8474-af74c9317d6b_isrForCrosstalkSources_1031738_12.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3659335a-5e14-4db9-8474-af74c9317d6b_isrForCrosstalkSources_1031738_12.stderr parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash 5df97f4b061e175e50840480b464384e parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c7e44c07-57b1-407c-92c6-11f2238d2735_isrForCrosstalkSources_1031707_3.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c7e44c07-57b1-407c-92c6-11f2238d2735_isrForCrosstalkSources_1031707_3.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 900df9f6cfdb68d433a4bd82970baf8a parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/07fa0ffe-4db0-4a0c-9015-a1e155b67ae5_isrForCrosstalkSources_1031706_13.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/07fa0ffe-4db0-4a0c-9015-a1e155b67ae5_isrForCrosstalkSources_1031706_13.stderr parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 552b43acf457003e098ed406f5a4241f parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/07261fda-ad71-4f72-bb63-4af523626fba_isrForCrosstalkSources_1031724_61.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/07261fda-ad71-4f72-bb63-4af523626fba_isrForCrosstalkSources_1031724_61.stderr parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash df2e46ceee01f9a651ec4151fdc95a6f parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/202dd593-41ad-4c7a-88a8-3001b7018776_isrForCrosstalkSources_1031711_44.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/202dd593-41ad-4c7a-88a8-3001b7018776_isrForCrosstalkSources_1031711_44.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash 13e21675d6654725717f906a5a71b91b parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/fce4cdb5-ddab-416a-a40c-60b8f86f7fc0_isrForCrosstalkSources_1031704_39.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/fce4cdb5-ddab-416a-a40c-60b8f86f7fc0_isrForCrosstalkSources_1031704_39.stderr parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash 8dd46afb07be01ab3d5473f7103347d9 parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/9e71acac-ae25-4952-965d-e559a40d009d_isrForCrosstalkSources_1031702_51.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/9e71acac-ae25-4952-965d-e559a40d009d_isrForCrosstalkSources_1031702_51.stderr parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash 428fb5a8e203cf1d2b7e4e4a9f00308f parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/d39377b4-851c-4642-aea6-582a07c69914_isrForCrosstalkSources_1031723_61.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/d39377b4-851c-4642-aea6-582a07c69914_isrForCrosstalkSources_1031723_61.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 49a25de4fe084110abf7d808574f0e35 parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bbd295d8-d986-401e-a418-2c8841cbaa3b_isrForCrosstalkSources_1031734_20.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bbd295d8-d986-401e-a418-2c8841cbaa3b_isrForCrosstalkSources_1031734_20.stderr parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash dd0e775fe320c13d9342327d731f9105 parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/2a0697a5-1421-41fd-b448-e51f4881bb6c_isrForCrosstalkSources_1031719_24.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/2a0697a5-1421-41fd-b448-e51f4881bb6c_isrForCrosstalkSources_1031719_24.stderr parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash 897b60bffe047f2009267e1218ff7716 parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/627c9a24-29ee-40bf-a852-733c1c57854f_isrForCrosstalkSources_1031715_49.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/627c9a24-29ee-40bf-a852-733c1c57854f_isrForCrosstalkSources_1031715_49.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash 93916435bac7e8b7b0d2f5c097006b30 parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/080cfe3e-b47f-4265-a4ca-f7f9fcbb1838_isrForCrosstalkSources_1031724_12.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/080cfe3e-b47f-4265-a4ca-f7f9fcbb1838_isrForCrosstalkSources_1031724_12.stderr parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash fc0383171cb27e30bd4bf3a8ef4efb14 parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6a0c0a16-e9ca-4486-9e0a-e73b8d13871a_isrForCrosstalkSources_1031711_34.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6a0c0a16-e9ca-4486-9e0a-e73b8d13871a_isrForCrosstalkSources_1031711_34.stderr parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash f0b18f557ff0f131822b2aed1f31a743 parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/991d3a8d-54c0-4553-96cc-8fd868a7c0a9_isrForCrosstalkSources_1031699_48.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/991d3a8d-54c0-4553-96cc-8fd868a7c0a9_isrForCrosstalkSources_1031699_48.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 049e377eda42bda8d963de6d582f8fc3 parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/d9168d6a-3af6-40e8-af4b-3514ed7be3f2_isrForCrosstalkSources_1031710_43.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/d9168d6a-3af6-40e8-af4b-3514ed7be3f2_isrForCrosstalkSources_1031710_43.stderr parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash 51a03d5f6df854338ec95d1918115a05 parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c164b655-d69d-49f3-94e3-39a2faa55ba3_isrForCrosstalkSources_1031726_3.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c164b655-d69d-49f3-94e3-39a2faa55ba3_isrForCrosstalkSources_1031726_3.stderr parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash 31d536d1e3b6c0c91c68dd90f4c846de parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/30c38e7a-31f8-4daf-9093-b1b7fb3f5de8_isrForCrosstalkSources_1031735_54.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/30c38e7a-31f8-4daf-9093-b1b7fb3f5de8_isrForCrosstalkSources_1031735_54.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash a9afbf19bbca6285949b365ab47ce6f8 parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1337471b-a2ac-4e25-8870-87d2b75d9a41_isrForCrosstalkSources_1031735_44.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1337471b-a2ac-4e25-8870-87d2b75d9a41_isrForCrosstalkSources_1031735_44.stderr parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash b5d1b5691e11f82d43b0fa860e254125 parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/c732f051-2857-4661-952b-059b5a5ef837_isrForCrosstalkSources_1031711_58.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/c732f051-2857-4661-952b-059b5a5ef837_isrForCrosstalkSources_1031711_58.stderr parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 13e87943a786a92916b47afe2c120405 parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2fd805e1-911e-4a26-8773-ebf2fea5fa35_isrForCrosstalkSources_1031731_62.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2fd805e1-911e-4a26-8773-ebf2fea5fa35_isrForCrosstalkSources_1031731_62.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 0ce2e9756fd6c7d99dcc0191c9501aa6 parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ad117bcc-0838-433b-993e-563a9ef4116f_isrForCrosstalkSources_1031731_56.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ad117bcc-0838-433b-993e-563a9ef4116f_isrForCrosstalkSources_1031731_56.stderr parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash 94259cced131fbb64cb4519337faf92e parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/eab76665-f406-4592-9cd3-ee89d2d41ea7_isrForCrosstalkSources_1031701_38.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/eab76665-f406-4592-9cd3-ee89d2d41ea7_isrForCrosstalkSources_1031701_38.stderr parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash a780e4a36211985332f5f54ec19c1410 parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/032fda4f-5e9f-454f-a28e-8c3ab1ed60fd_isrForCrosstalkSources_1031713_18.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/032fda4f-5e9f-454f-a28e-8c3ab1ed60fd_isrForCrosstalkSources_1031713_18.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash a3bbb7974022adc343addbbd09014660 parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/49e67eef-5289-476b-b05b-c54628a7d016_isrForCrosstalkSources_1031724_37.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/49e67eef-5289-476b-b05b-c54628a7d016_isrForCrosstalkSources_1031724_37.stderr parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash ccc27f7fba65d902eb31e1a23d334698 parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/82a9fc08-279b-4ea2-ac0a-ddeaf37bd650_isrForCrosstalkSources_1031732_46.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/82a9fc08-279b-4ea2-ac0a-ddeaf37bd650_isrForCrosstalkSources_1031732_46.stderr parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 65900c23717323f36f5fc6f9b3973935 parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/93968d73-4d2d-481d-a12e-8cb77c9f0a82_isrForCrosstalkSources_1031702_50.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/93968d73-4d2d-481d-a12e-8cb77c9f0a82_isrForCrosstalkSources_1031702_50.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 2d6371d0c6d24a66325a5b04ddf2e047 parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/26035f50-f7ab-4da8-95cf-ca6352c87e4a_isrForCrosstalkSources_1031733_3.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/26035f50-f7ab-4da8-95cf-ca6352c87e4a_isrForCrosstalkSources_1031733_3.stderr parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash 8f2fa0563fc2bb07ade144243dda1b1e parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/649e1403-ffd1-41af-9c44-5899eff97e0c_isrForCrosstalkSources_1031733_41.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/649e1403-ffd1-41af-9c44-5899eff97e0c_isrForCrosstalkSources_1031733_41.stderr parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash ec8c811a571429bca20aaa4cd985acf3 parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ea812b46-975d-416f-a48f-0e3345807ff1_isrForCrosstalkSources_1031711_31.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ea812b46-975d-416f-a48f-0e3345807ff1_isrForCrosstalkSources_1031711_31.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 69225de1998ef1e71568282a5795d80b parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:07 proc_lsst.multi:170 [INFO] [multi] job_id 17147943 proc_lsst.multi INFO: [multi] job_id 17147943 2024-03-17 04:02:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-17 04:02:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147943 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147943 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17147943 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-17 04:02:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-17 04:02:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-17 04:02:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e79f8233-62f1-4fb8-98bf-c7c5e6796daa_isrForCrosstalkSources_1031731_61.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e79f8233-62f1-4fb8-98bf-c7c5e6796daa_isrForCrosstalkSources_1031731_61.stderr parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash d559df4e14931168327be078ff6360d6 parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd84c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/34e48b3d-1eea-4ef6-b8f1-6efcb43c93ee_isrForCrosstalkSources_1031735_61.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/34e48b3d-1eea-4ef6-b8f1-6efcb43c93ee_isrForCrosstalkSources_1031735_61.stderr parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash 1891bbc9e05e026e189155d3ea2c7d52 parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd87f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6b2f5314-2be9-4661-920d-c49aae9f1892_isrForCrosstalkSources_1031716_14.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6b2f5314-2be9-4661-920d-c49aae9f1892_isrForCrosstalkSources_1031716_14.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash 950cf36b4a26d714cd8675edf6da78f3 parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd86020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/e9e8b19f-ec80-43ae-ac6f-efc80d7d8e0a_isrForCrosstalkSources_1031724_41.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/e9e8b19f-ec80-43ae-ac6f-efc80d7d8e0a_isrForCrosstalkSources_1031724_41.stderr parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash c950a1de8f776cf738fd0847d51d8e82 parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/aedc4585-34ca-447a-80b0-917eea51e7bf_isrForCrosstalkSources_1031700_54.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/aedc4585-34ca-447a-80b0-917eea51e7bf_isrForCrosstalkSources_1031700_54.stderr parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash ab84a578e25936c51ceaefd8ad6bd5a5 parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/f30b31a7-9b3a-4d90-bca2-ac9ffd273efd_isrForCrosstalkSources_1031708_36.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/f30b31a7-9b3a-4d90-bca2-ac9ffd273efd_isrForCrosstalkSources_1031708_36.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash 4ea8a3572b228b290fe342587a0d2a0a parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/63fd52fd-25cc-4156-bd17-c6517aa3c0f3_isrForCrosstalkSources_1031712_38.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/63fd52fd-25cc-4156-bd17-c6517aa3c0f3_isrForCrosstalkSources_1031712_38.stderr parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash 55cb3a8a245bb1be98ad9b8ec8c39e46 parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2e83e6a2-2ee9-4dd7-a427-7ed315957cfb_isrForCrosstalkSources_1031736_30.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2e83e6a2-2ee9-4dd7-a427-7ed315957cfb_isrForCrosstalkSources_1031736_30.stderr parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash b4ced211059ee7bc93ad986a60873d9c parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/beea3b8e-6cee-4ee1-b9c9-5125125bd487_isrForCrosstalkSources_1031701_53.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/beea3b8e-6cee-4ee1-b9c9-5125125bd487_isrForCrosstalkSources_1031701_53.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash d75d09fe8d3f5c04e5343e5db09a2f07 parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdceac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8f389777-933f-49da-8952-e17bd2c65cce_isrForCrosstalkSources_1031732_61.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8f389777-933f-49da-8952-e17bd2c65cce_isrForCrosstalkSources_1031732_61.stderr parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash 63469c8b9271018caccf1baa87e14df7 parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdceca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/85029dc4-d5b8-4117-a51f-b8b2bb2562ea_isrForCrosstalkSources_1031707_40.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/85029dc4-d5b8-4117-a51f-b8b2bb2562ea_isrForCrosstalkSources_1031707_40.stderr parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash ce027fc6f842501cfb350b495ce78867 parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/33b2f4ef-6f99-45e8-8e05-c8957e01fc7a_isrForCrosstalkSources_1031709_42.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/33b2f4ef-6f99-45e8-8e05-c8957e01fc7a_isrForCrosstalkSources_1031709_42.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 397f4cac8edf6d9f4d8f0d9f3f7f8ca2 parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdced40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d0bdd09e-5a18-4264-9227-dffc9d379879_isrForCrosstalkSources_1031730_23.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d0bdd09e-5a18-4264-9227-dffc9d379879_isrForCrosstalkSources_1031730_23.stderr parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash 9b47e878f769e75637b511d3288117a9 parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdceb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/f341cd9f-9072-4cb8-af7a-3d949bc1a5d5_isrForCrosstalkSources_1031724_58.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/f341cd9f-9072-4cb8-af7a-3d949bc1a5d5_isrForCrosstalkSources_1031724_58.stderr parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 942 has memoization hash 38c2e5b0a6f54718551aa454f67097f3 parsl.dataflow.memoization INFO: Task 942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/222d355a-e1ab-4be2-b9db-f5d218caccbf_isrForCrosstalkSources_1031725_40.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/222d355a-e1ab-4be2-b9db-f5d218caccbf_isrForCrosstalkSources_1031725_40.stderr parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 140c21f80884bdf7dd19b9fa5423d4a7 parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/fa69d29d-9890-41a0-a319-529db4917e9a_isrForCrosstalkSources_1031721_12.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/fa69d29d-9890-41a0-a319-529db4917e9a_isrForCrosstalkSources_1031721_12.stderr parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash f46e6a80f47b4714ecb0ab4c00cf0550 parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a5a9b035-5e52-4643-bff3-f081c90e799a_isrForCrosstalkSources_1031728_6.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a5a9b035-5e52-4643-bff3-f081c90e799a_isrForCrosstalkSources_1031728_6.stderr parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash b909b62f3ecf1674f7868a70f4b11757 parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/6400c591-3774-4497-a0c3-eda795fec70e_isrForCrosstalkSources_1031706_32.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/6400c591-3774-4497-a0c3-eda795fec70e_isrForCrosstalkSources_1031706_32.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash 00204200000aea0d31a082e74d3994a3 parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e898f0e0-743a-4c76-8a72-3626fc4167ff_isrForCrosstalkSources_1031709_4.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e898f0e0-743a-4c76-8a72-3626fc4167ff_isrForCrosstalkSources_1031709_4.stderr parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash c7ca833d3cc732cd11468dcbf4f0434f parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/9bb5b6c8-d08b-442a-b44b-b38c5a34ca72_isrForCrosstalkSources_1031735_51.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/9bb5b6c8-d08b-442a-b44b-b38c5a34ca72_isrForCrosstalkSources_1031735_51.stderr parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 571ad006e1b22ce70ac5ba896765e209 parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/04360d5e-019e-4539-8afd-e5c3782c6223_isrForCrosstalkSources_1031715_53.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/04360d5e-019e-4539-8afd-e5c3782c6223_isrForCrosstalkSources_1031715_53.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash 04d64e7d1bae057de45e73651d13123b parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/26906c93-5297-4d21-b505-8dcf71f6a2df_isrForCrosstalkSources_1031713_45.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/26906c93-5297-4d21-b505-8dcf71f6a2df_isrForCrosstalkSources_1031713_45.stderr parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 950 has memoization hash 66c953155bb47356654e8e8b91dae457 parsl.dataflow.memoization INFO: Task 950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/990f8c89-f19b-4bb7-a3d1-35a94173ae13_isrForCrosstalkSources_1031730_48.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/990f8c89-f19b-4bb7-a3d1-35a94173ae13_isrForCrosstalkSources_1031730_48.stderr parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash 4553f923192edae12676ed9891bff432 parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/091f5583-93e3-4a9b-b98b-7b488d7e371c_isrForCrosstalkSources_1031719_42.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/091f5583-93e3-4a9b-b98b-7b488d7e371c_isrForCrosstalkSources_1031719_42.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 036b3d390d90b3464c2b1728b0f19783 parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4c5c9a89-931e-4b67-a3a0-17cff2e290ae_isrForCrosstalkSources_1031732_35.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4c5c9a89-931e-4b67-a3a0-17cff2e290ae_isrForCrosstalkSources_1031732_35.stderr parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash 843c21253db619eda2781f4e42666e18 parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/8649c994-df28-40e4-9d50-c8365816e4c2_isrForCrosstalkSources_1031727_45.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/8649c994-df28-40e4-9d50-c8365816e4c2_isrForCrosstalkSources_1031727_45.stderr parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 24295cc9de4eef21c512f759597409ac parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c76310df-2ffc-403c-9f05-69373879a9f3_isrForCrosstalkSources_1031725_29.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c76310df-2ffc-403c-9f05-69373879a9f3_isrForCrosstalkSources_1031725_29.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash 4764649b2d23c5be4eff75fdb62d583c parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5366fd81-c367-458d-a8c6-0dea2f97bf88_isrForCrosstalkSources_1031703_54.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5366fd81-c367-458d-a8c6-0dea2f97bf88_isrForCrosstalkSources_1031703_54.stderr parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash 839cc6871b2eecadb78243d6476636d8 parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/dceac69a-183d-4a35-80ab-5119b4a87a27_isrForCrosstalkSources_1031730_8.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/dceac69a-183d-4a35-80ab-5119b4a87a27_isrForCrosstalkSources_1031730_8.stderr parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash 81563c38bc518487bd2f3c4d123ae236 parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/680488e0-c568-426e-9398-efa51a89f87d_isrForCrosstalkSources_1031725_44.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/680488e0-c568-426e-9398-efa51a89f87d_isrForCrosstalkSources_1031725_44.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash c79c2f0bd71d233f9febe89d0d38b175 parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9aa4793e-e063-4e19-b325-2ae0e41b5b19_isrForCrosstalkSources_1031733_46.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9aa4793e-e063-4e19-b325-2ae0e41b5b19_isrForCrosstalkSources_1031733_46.stderr parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash ea9c187629dead6baff764c7d59523cd parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e5791306-2b01-45a6-a20b-729a3b81de9f_isrForCrosstalkSources_1031710_46.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e5791306-2b01-45a6-a20b-729a3b81de9f_isrForCrosstalkSources_1031710_46.stderr parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash e24a5c030cb0236ccce57bf5a02d89e3 parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f075f619-1266-48f7-a512-cabcc679c417_isrForCrosstalkSources_1031720_2.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f075f619-1266-48f7-a512-cabcc679c417_isrForCrosstalkSources_1031720_2.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 617b9f16a532d963036c2d12ac567f2b parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2f069e2b-9335-465a-a28f-f8d8a4dbf935_isrForCrosstalkSources_1031703_29.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2f069e2b-9335-465a-a28f-f8d8a4dbf935_isrForCrosstalkSources_1031703_29.stderr parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash 021d8922db0ae05bf92025be5ea9b5f8 parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/439fce95-a1e9-46b8-96b2-6d87776d364f_isrForCrosstalkSources_1031716_57.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/439fce95-a1e9-46b8-96b2-6d87776d364f_isrForCrosstalkSources_1031716_57.stderr parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash f620fbf1bbf3b37839dd57da7bbf0e6d parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/6ec9ebf4-add5-46e6-a37a-e4fc1bdff4d8_isrForCrosstalkSources_1031736_39.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/6ec9ebf4-add5-46e6-a37a-e4fc1bdff4d8_isrForCrosstalkSources_1031736_39.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash ee65648a2cffca0de1e47bdf5f58279e parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/891e1f80-6e1d-40a2-a092-1d744333e6ff_isrForCrosstalkSources_1031734_44.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/891e1f80-6e1d-40a2-a092-1d744333e6ff_isrForCrosstalkSources_1031734_44.stderr parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 965 has memoization hash 56e9ba392cf4d3fd63350d559bc90c1c parsl.dataflow.memoization INFO: Task 965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/cc77e4f3-fe1c-4d8a-9b2e-bf33ab88ffbb_isrForCrosstalkSources_1031711_18.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/cc77e4f3-fe1c-4d8a-9b2e-bf33ab88ffbb_isrForCrosstalkSources_1031711_18.stderr parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash b0a3c563418b5ff391fa4c4d7e318ee4 parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/84171944-cec1-4b9e-bbf4-35b8654f0f38_isrForCrosstalkSources_1031705_34.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/84171944-cec1-4b9e-bbf4-35b8654f0f38_isrForCrosstalkSources_1031705_34.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash 320808c182217785b23536e5b05d487b parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/97497fc1-37da-4fa3-8a5f-bcdc2ef5cf0e_isrForCrosstalkSources_1031717_9.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/97497fc1-37da-4fa3-8a5f-bcdc2ef5cf0e_isrForCrosstalkSources_1031717_9.stderr parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash a55f02889bff4a443225d9d929079579 parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/283a6368-6d93-4cb5-bde1-85506dd006a0_isrForCrosstalkSources_1031736_43.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/283a6368-6d93-4cb5-bde1-85506dd006a0_isrForCrosstalkSources_1031736_43.stderr parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash 1a70af98cd3bc3fde53a6329ed10e29d parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/33bf5146-4d7f-4eed-9c23-965f98a414e7_isrForCrosstalkSources_1031721_35.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/33bf5146-4d7f-4eed-9c23-965f98a414e7_isrForCrosstalkSources_1031721_35.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash 7ef2f849faff86224a3d8d3fe785589f parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/3030444e-8ce1-4059-bb65-b99ef1844987_isrForCrosstalkSources_1031733_50.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/3030444e-8ce1-4059-bb65-b99ef1844987_isrForCrosstalkSources_1031733_50.stderr parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash 4f9d153df79333922337a2a046c72146 parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc440e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/6c046e9d-d89a-4b88-b156-c0e1bc597007_isrForCrosstalkSources_1031731_32.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/6c046e9d-d89a-4b88-b156-c0e1bc597007_isrForCrosstalkSources_1031731_32.stderr parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 9b9583742e92c17eda4955ff1842c362 parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/64c1863e-046e-4103-955e-9ab0ae3aa435_isrForCrosstalkSources_1031707_35.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/64c1863e-046e-4103-955e-9ab0ae3aa435_isrForCrosstalkSources_1031707_35.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash ee0cff02c8757dbb0edb079c3d1d3133 parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/544a33b2-f1a8-462b-abb3-dd385ebc1af9_isrForCrosstalkSources_1031721_19.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/544a33b2-f1a8-462b-abb3-dd385ebc1af9_isrForCrosstalkSources_1031721_19.stderr parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash 2fccad5f96652bd7d78f9b5b630cc660 parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc442c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1d445ddc-c885-4a59-958a-a446a82a599a_isrForCrosstalkSources_1031733_53.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1d445ddc-c885-4a59-958a-a446a82a599a_isrForCrosstalkSources_1031733_53.stderr parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash c110d4f4c0eda7144f12d90c0d30158f parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d48cfc72-f424-476a-8fe8-503c9def50e4_isrForCrosstalkSources_1031738_59.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d48cfc72-f424-476a-8fe8-503c9def50e4_isrForCrosstalkSources_1031738_59.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 7c559b0006c8f935a1e26df2941ccbf2 parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f2c9145d-f822-44dd-9310-20eafccda547_isrForCrosstalkSources_1031721_45.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f2c9145d-f822-44dd-9310-20eafccda547_isrForCrosstalkSources_1031721_45.stderr parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash d6db4d54cc7b20b34d4388e0bb0b1682 parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c0d80edd-f7a0-4652-b51f-a03f1a5d79b2_isrForCrosstalkSources_1031727_48.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c0d80edd-f7a0-4652-b51f-a03f1a5d79b2_isrForCrosstalkSources_1031727_48.stderr parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash 90214b810d089dd7e65188e36b80a394 parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc444a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7d32467c-e089-4318-8aeb-8a02ba447397_isrForCrosstalkSources_1031702_14.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7d32467c-e089-4318-8aeb-8a02ba447397_isrForCrosstalkSources_1031702_14.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash 5b6b3ab7775ec7c713c5a6b36db15956 parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc445e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/392913c0-ae88-42b2-9e12-727daf5976e3_isrForCrosstalkSources_1031700_14.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/392913c0-ae88-42b2-9e12-727daf5976e3_isrForCrosstalkSources_1031700_14.stderr parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 184c2974c077e07de341fc3b1e7ea40a parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/40cd924e-ff8d-4c1b-a468-669d7618eb9a_isrForCrosstalkSources_1031710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/40cd924e-ff8d-4c1b-a468-669d7618eb9a_isrForCrosstalkSources_1031710_61.stderr parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash 22beb60820b4861e115218d981ed4629 parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ee00135e-4446-439e-9f55-ee493a0bef58_isrForCrosstalkSources_1031737_8.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ee00135e-4446-439e-9f55-ee493a0bef58_isrForCrosstalkSources_1031737_8.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash ccd97cfe36d2ce6ee3a7b9e6c2e2f130 parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/9846362d-eed9-475b-96b3-2a40d0b87187_isrForCrosstalkSources_1031717_45.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/9846362d-eed9-475b-96b3-2a40d0b87187_isrForCrosstalkSources_1031717_45.stderr parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash 3b76fe8049d8e7bea1e1d9858348db20 parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc447c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/73805eb4-e0a3-4681-905f-4d52e844f866_isrForCrosstalkSources_1031700_33.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/73805eb4-e0a3-4681-905f-4d52e844f866_isrForCrosstalkSources_1031700_33.stderr parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash 4c73ae4c2289c1b7465954b3e09612fe parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/eebe71aa-974c-4b22-817b-ff6ead4e9341_isrForCrosstalkSources_1031717_54.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/eebe71aa-974c-4b22-817b-ff6ead4e9341_isrForCrosstalkSources_1031717_54.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash a586ba5f383d417f7a581a919a227b38 parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc449a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/b59921bd-e26a-49cd-affa-1f0e3cd99df7_isrForCrosstalkSources_1031737_11.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/b59921bd-e26a-49cd-affa-1f0e3cd99df7_isrForCrosstalkSources_1031737_11.stderr parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash 4d4a2d02e123b558b8a3a5337fa4dae7 parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b6eb666b-4282-4a70-b3fc-fe957fb918e1_isrForCrosstalkSources_1031721_2.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b6eb666b-4282-4a70-b3fc-fe957fb918e1_isrForCrosstalkSources_1031721_2.stderr parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash aad107ca4fd231c93ae69ab7d4b2fea0 parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/9adcaf62-5442-4ebb-8299-58ab3bc4b694_isrForCrosstalkSources_1031727_56.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/9adcaf62-5442-4ebb-8299-58ab3bc4b694_isrForCrosstalkSources_1031727_56.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 988 has memoization hash 1d3f04f73e4b91fcf38fa055fefc63bc parsl.dataflow.memoization INFO: Task 988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/aa7b7896-72e3-4eae-a71e-0d47619c36fb_isrForCrosstalkSources_1031712_39.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/aa7b7896-72e3-4eae-a71e-0d47619c36fb_isrForCrosstalkSources_1031712_39.stderr parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash 77bf099e75700ae99bb7275f85f689a5 parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/8a26e547-6fe3-45ff-8304-6d08dd5d7c84_isrForCrosstalkSources_1031705_45.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/8a26e547-6fe3-45ff-8304-6d08dd5d7c84_isrForCrosstalkSources_1031705_45.stderr parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash ff65239a5d27ae3040cc66e6f7bfa166 parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/27eb1a40-da90-47cc-9ad3-7b5b3a9b6e35_isrForCrosstalkSources_1031707_52.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/27eb1a40-da90-47cc-9ad3-7b5b3a9b6e35_isrForCrosstalkSources_1031707_52.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 9ea4f7647e189c5cddb57b6aa7b8dd11 parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/38a5ab9c-33b3-4bb1-8c70-bbc97ddecf09_isrForCrosstalkSources_1031706_14.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/38a5ab9c-33b3-4bb1-8c70-bbc97ddecf09_isrForCrosstalkSources_1031706_14.stderr parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash 9a36faa83144c09908e08cb82ebfdba0 parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/78d7105f-ff93-4999-a6fa-b7879a1d3fbd_isrForCrosstalkSources_1031716_13.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/78d7105f-ff93-4999-a6fa-b7879a1d3fbd_isrForCrosstalkSources_1031716_13.stderr parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 31304960c0506db8cf9579105eaee9fd parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/b2335663-c8aa-47aa-b96b-da8708296f2a_isrForCrosstalkSources_1031700_45.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/b2335663-c8aa-47aa-b96b-da8708296f2a_isrForCrosstalkSources_1031700_45.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 105d37ea355a35635464efba606dbf2b parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/dc41d75c-ae66-46b4-8da4-c88c347c36d6_isrForCrosstalkSources_1031712_8.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/dc41d75c-ae66-46b4-8da4-c88c347c36d6_isrForCrosstalkSources_1031712_8.stderr parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 86efb062986353043f5518eca5dd2304 parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fc7359d7-2576-4355-a30c-d187172e51c8_isrForCrosstalkSources_1031723_62.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fc7359d7-2576-4355-a30c-d187172e51c8_isrForCrosstalkSources_1031723_62.stderr parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash b1ce765a9900d8ee069bedca9de443c9 parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eb97c069-f42d-4092-84a7-78765bc627e6_isrForCrosstalkSources_1031719_12.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eb97c069-f42d-4092-84a7-78765bc627e6_isrForCrosstalkSources_1031719_12.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash 33b72f64c0cd3b8fe0fda54eea83025d parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc451c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:08 proc_lsst.multi:170 [INFO] [multi] job_id 17147944 proc_lsst.multi INFO: [multi] job_id 17147944 2024-03-17 04:02:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-17 04:02:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147944 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147944 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17147944 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-17 04:02:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-17 04:02:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-17 04:02:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/9f07a476-3127-47ec-a3e0-fa77740d4d2a_isrForCrosstalkSources_1031715_40.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/9f07a476-3127-47ec-a3e0-fa77740d4d2a_isrForCrosstalkSources_1031715_40.stderr parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash 7e4a33a2c97535bb120df6b638d9ebfa parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdcff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/96089947-a339-4bb9-8d16-4edbfaa82ec6_isrForCrosstalkSources_1031735_6.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/96089947-a339-4bb9-8d16-4edbfaa82ec6_isrForCrosstalkSources_1031735_6.stderr parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash 315b87538743c5d1b1e0b4916255146e parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/dc10ffe5-073e-4189-b2f7-f0eb5b96e036_isrForCrosstalkSources_1031737_38.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/dc10ffe5-073e-4189-b2f7-f0eb5b96e036_isrForCrosstalkSources_1031737_38.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash d1f352cdaf6f1ead9f254be7fbfdbbd5 parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fdce7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1d2894bb-f2fa-43d6-9f49-8222a7b50e03_isrForCrosstalkSources_1031703_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1d2894bb-f2fa-43d6-9f49-8222a7b50e03_isrForCrosstalkSources_1031703_16.stderr parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash ccfd50ed08211b6da22d8116e9099fbe parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/562efe41-c410-4ae5-93b4-b306a512f584_isrForCrosstalkSources_1031735_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/562efe41-c410-4ae5-93b4-b306a512f584_isrForCrosstalkSources_1031735_46.stderr parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash c6d3d62ed993055cef617568839833f4 parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc454e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6d69e64d-4227-47f4-a442-300b825f6d09_isrForCrosstalkSources_1031733_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6d69e64d-4227-47f4-a442-300b825f6d09_isrForCrosstalkSources_1031733_61.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash e5483a4e99bf45f8ec9bb7d5857d2c45 parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/478f88f1-e8b8-409b-823a-95d70337e7eb_isrForCrosstalkSources_1031723_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/478f88f1-e8b8-409b-823a-95d70337e7eb_isrForCrosstalkSources_1031723_34.stderr parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash 0bc363971e2c01b051ca5a85c83a8ba6 parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/1690753d-9bac-4268-a70c-16b8ae3bd993_isrForCrosstalkSources_1031737_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/1690753d-9bac-4268-a70c-16b8ae3bd993_isrForCrosstalkSources_1031737_9.stderr parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash 7e431e1de491620a1c5807372b674ee5 parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/84416d15-14ec-4fb1-813d-d528a6874f33_isrForCrosstalkSources_1031720_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/84416d15-14ec-4fb1-813d-d528a6874f33_isrForCrosstalkSources_1031720_6.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash 08c1f1d1f6417cc046af17a56d5dfcb3 parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc456c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/8c028075-1abb-4f72-a657-a0f0b94c112f_isrForCrosstalkSources_1031710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/8c028075-1abb-4f72-a657-a0f0b94c112f_isrForCrosstalkSources_1031710_31.stderr parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash c08b43d6e1086bc6d39c7e9e919ad37b parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0813cf5c-6429-45f0-8689-54ec82f75d68_isrForCrosstalkSources_1031720_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0813cf5c-6429-45f0-8689-54ec82f75d68_isrForCrosstalkSources_1031720_10.stderr parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash d003d40ea1aff436f7e07b53c7f9e421 parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/810c069f-842f-4783-97c1-1188943670b2_isrForCrosstalkSources_1031700_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/810c069f-842f-4783-97c1-1188943670b2_isrForCrosstalkSources_1031700_9.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash 498e347c7e689095e1d4f5b549e341b9 parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5e8c5b82-1466-4fd4-91e9-938f4c744e32_isrForCrosstalkSources_1031711_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5e8c5b82-1466-4fd4-91e9-938f4c744e32_isrForCrosstalkSources_1031711_61.stderr parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash 6eb6b87e0a95c3785fbc1e8e13b01475 parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc458a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/da714196-201e-430e-8c9e-ad38ea0d9446_isrForCrosstalkSources_1031707_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/da714196-201e-430e-8c9e-ad38ea0d9446_isrForCrosstalkSources_1031707_44.stderr parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash 4f4ed311b0278f1b7b0dd6b309a986c7 parsl.dataflow.memoization INFO: Task 1011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f942457f-55ee-4f73-af7d-8c8478099116_isrForCrosstalkSources_1031709_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f942457f-55ee-4f73-af7d-8c8478099116_isrForCrosstalkSources_1031709_61.stderr parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash c22fbdfde1f03f80b939c86ce1c21c62 parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/757b4c92-c43e-41ff-8999-17ad223c3cd6_isrForCrosstalkSources_1031705_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/757b4c92-c43e-41ff-8999-17ad223c3cd6_isrForCrosstalkSources_1031705_6.stderr parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 766bedc8c926b37484748064705075ef parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/55dbe9c5-4e49-43aa-84d5-8d6f2f880f8e_isrForCrosstalkSources_1031724_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/55dbe9c5-4e49-43aa-84d5-8d6f2f880f8e_isrForCrosstalkSources_1031724_16.stderr parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash 89674ff2b10d62d401b6ba9736586f32 parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc459e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bb74893c-7394-414d-9ab0-785c06a1a25c_isrForCrosstalkSources_1031729_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bb74893c-7394-414d-9ab0-785c06a1a25c_isrForCrosstalkSources_1031729_34.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash 9482d93d3f840c1306524ef7c5f2affc parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/31b593b0-85be-4bed-bac5-1fb41bdd2d7e_isrForCrosstalkSources_1031713_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/31b593b0-85be-4bed-bac5-1fb41bdd2d7e_isrForCrosstalkSources_1031713_14.stderr parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash 5375831be53bc4731896cfad58daff67 parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/93137045-af91-400a-9b38-eaab3e8f2c83_isrForCrosstalkSources_1031702_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/93137045-af91-400a-9b38-eaab3e8f2c83_isrForCrosstalkSources_1031702_35.stderr parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash 8ae6b6bfb14f5cd7561a1bbafde19ce6 parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/159fccdf-829e-49c8-aaa8-8dc18b055eb5_isrForCrosstalkSources_1031707_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/159fccdf-829e-49c8-aaa8-8dc18b055eb5_isrForCrosstalkSources_1031707_55.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash 42b15466e693481635f2a75f29d94140 parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/06a36b25-665d-4c31-b61b-3c8359d96b72_isrForCrosstalkSources_1031708_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/06a36b25-665d-4c31-b61b-3c8359d96b72_isrForCrosstalkSources_1031708_49.stderr parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 8bfd3e9880dbcb6651907930fbcffc06 parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/baff6d74-0144-4be9-8681-ad75c7532d88_isrForCrosstalkSources_1031724_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/baff6d74-0144-4be9-8681-ad75c7532d88_isrForCrosstalkSources_1031724_55.stderr parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 5adeedd6099ce07c0dd52963d146d564 parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/537d0c6a-d244-4cb1-96e7-17b0aa4a4760_isrForCrosstalkSources_1031710_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/537d0c6a-d244-4cb1-96e7-17b0aa4a4760_isrForCrosstalkSources_1031710_18.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash e0d097faf6174f3e2b4dd5346f07359f parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f45af25f-77fe-4996-86cd-65224127a010_isrForCrosstalkSources_1031700_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f45af25f-77fe-4996-86cd-65224127a010_isrForCrosstalkSources_1031700_32.stderr parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 3530210d17ff1a2542094a1f735a9bee parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c6401ebf-03ae-4d1a-8af6-eba01a10440d_isrForCrosstalkSources_1031732_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c6401ebf-03ae-4d1a-8af6-eba01a10440d_isrForCrosstalkSources_1031732_19.stderr parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash 30a0e87cffc923a99b3ccb48dea62c81 parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/2f0fd5d4-b8e0-42c8-94c8-9baedd6cb6c6_isrForCrosstalkSources_1031718_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/2f0fd5d4-b8e0-42c8-94c8-9baedd6cb6c6_isrForCrosstalkSources_1031718_24.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 1b26bdbf4ade0afe9c06d0f36e7c28cd parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/86dfa5e7-cf16-4e02-9599-91c26e3a9ce2_isrForCrosstalkSources_1031737_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/86dfa5e7-cf16-4e02-9599-91c26e3a9ce2_isrForCrosstalkSources_1031737_13.stderr parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash deefca112cbae3909002a341cf25ec0f parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc462a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/67c2826a-0186-4f0a-bced-15a0ecff34ab_isrForCrosstalkSources_1031715_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/67c2826a-0186-4f0a-bced-15a0ecff34ab_isrForCrosstalkSources_1031715_4.stderr parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash 38c2420893a0e6ec4eeb1276d30a7760 parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc460c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/41beba85-719b-45b4-89ba-ddcfe84cbf6e_isrForCrosstalkSources_1031719_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/41beba85-719b-45b4-89ba-ddcfe84cbf6e_isrForCrosstalkSources_1031719_45.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash d100b113725cb743cba825fce98e05bb parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/1bec8cd7-9eda-43c6-9480-79ed701f8ee8_isrForCrosstalkSources_1031734_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/1bec8cd7-9eda-43c6-9480-79ed701f8ee8_isrForCrosstalkSources_1031734_3.stderr parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash 78a8ebec154da3e3c5f5aaebea486df0 parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc463e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/81835bc2-ea10-4bca-9a14-fb4d051fceb1_isrForCrosstalkSources_1031706_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/81835bc2-ea10-4bca-9a14-fb4d051fceb1_isrForCrosstalkSources_1031706_16.stderr parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash c79ee4f72db2ddf9b72155da93820ca0 parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7efc6b0e-2ce8-45d6-aacc-c951b441fe59_isrForCrosstalkSources_1031704_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7efc6b0e-2ce8-45d6-aacc-c951b441fe59_isrForCrosstalkSources_1031704_10.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 34ee2c97aee86f8088dbbc929980f96a parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc465c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/1e1482d4-d51f-40a2-aa51-8b8823ea4815_isrForCrosstalkSources_1031728_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/1e1482d4-d51f-40a2-aa51-8b8823ea4815_isrForCrosstalkSources_1031728_39.stderr parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash 5c14441a8fa25be68868f8f4ad9156d4 parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/0c8274e9-3fd0-4602-aed8-687519b63d1d_isrForCrosstalkSources_1031738_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/0c8274e9-3fd0-4602-aed8-687519b63d1d_isrForCrosstalkSources_1031738_62.stderr parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash ea8f61dc79f49e19141682d6ede662d0 parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/159a0800-55b8-4561-a1a4-547b6bc20224_isrForCrosstalkSources_1031721_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/159a0800-55b8-4561-a1a4-547b6bc20224_isrForCrosstalkSources_1031721_52.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash d91d59ca2ea130a1737c9d53a009e668 parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fde7526e-92f0-492a-8faa-3dcdab7a63a8_isrForCrosstalkSources_1031729_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fde7526e-92f0-492a-8faa-3dcdab7a63a8_isrForCrosstalkSources_1031729_17.stderr parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash b2fbf46e70993a059145f7fecbb22ec7 parsl.dataflow.memoization INFO: Task 1034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc467a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c11a230f-a8dd-4253-a521-47dc6dc2da6f_isrForCrosstalkSources_1031732_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c11a230f-a8dd-4253-a521-47dc6dc2da6f_isrForCrosstalkSources_1031732_53.stderr parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 5016f2c5496430593e9ecc0a96ee5acf parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc468e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/26dedbba-e4ca-41b5-a81a-da7393bc1f87_isrForCrosstalkSources_1031721_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/26dedbba-e4ca-41b5-a81a-da7393bc1f87_isrForCrosstalkSources_1031721_47.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash 2911c03442b5d689c47d325c23dd4f7f parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/4c1e4e4b-6977-4306-b210-90f57c78b4f5_isrForCrosstalkSources_1031700_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/4c1e4e4b-6977-4306-b210-90f57c78b4f5_isrForCrosstalkSources_1031700_20.stderr parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 558e0a65dc0adf616e318ddfba2801b7 parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/abe372b6-7f14-4322-a586-72e4793cb40d_isrForCrosstalkSources_1031708_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/abe372b6-7f14-4322-a586-72e4793cb40d_isrForCrosstalkSources_1031708_29.stderr parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash b92b6a32b8298f55ca880ceba0bc684c parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1763f9e8-a377-40d2-a59a-2318b437a6b3_isrForCrosstalkSources_1031706_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1763f9e8-a377-40d2-a59a-2318b437a6b3_isrForCrosstalkSources_1031706_59.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash c53b0086550bb3c7922568238d14dbef parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/bfbf3363-c933-4b29-aec8-7991048053d4_isrForCrosstalkSources_1031727_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/bfbf3363-c933-4b29-aec8-7991048053d4_isrForCrosstalkSources_1031727_9.stderr parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash 57c5b7bc853e5bf03c1910c9cc71ec10 parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/f4d4ed5c-3640-4c19-867f-a3af103aec50_isrForCrosstalkSources_1031701_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/f4d4ed5c-3640-4c19-867f-a3af103aec50_isrForCrosstalkSources_1031701_49.stderr parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 91be56eed9a7b77b813b53fc9824edb8 parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/5bf2a9b2-eec9-4a06-b724-cb4c42c49d10_isrForCrosstalkSources_1031710_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/5bf2a9b2-eec9-4a06-b724-cb4c42c49d10_isrForCrosstalkSources_1031710_55.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 8c77b3dab1cb09fd291c6753bb0f11f8 parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4a5a7275-514c-4a80-90f4-5c9b19f4594b_isrForCrosstalkSources_1031719_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4a5a7275-514c-4a80-90f4-5c9b19f4594b_isrForCrosstalkSources_1031719_61.stderr parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash fc0ba5f3bc53584087cf12b7a81ca1b8 parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/caef9003-7236-41f8-9f68-4de6cfd65d6c_isrForCrosstalkSources_1031719_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/caef9003-7236-41f8-9f68-4de6cfd65d6c_isrForCrosstalkSources_1031719_11.stderr parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash ffa27af2d2d3e702a86351aba407fde1 parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/53382859-0a6f-4d96-9c5f-41d243e6409a_isrForCrosstalkSources_1031716_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/53382859-0a6f-4d96-9c5f-41d243e6409a_isrForCrosstalkSources_1031716_53.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash 9b6218cbb4720dfb539d547feeefa021 parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/acbe27f6-cda5-43e2-be00-ed8d1529ae22_isrForCrosstalkSources_1031699_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/acbe27f6-cda5-43e2-be00-ed8d1529ae22_isrForCrosstalkSources_1031699_9.stderr parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 0ee479a8fe50f03a0214d85984e6cc3c parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7e6932e8-9188-449f-b947-c641e38d4845_isrForCrosstalkSources_1031729_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7e6932e8-9188-449f-b947-c641e38d4845_isrForCrosstalkSources_1031729_29.stderr parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash ae078d042eb5c2f21358aad3cec60fec parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/57efe522-1e5c-496b-a379-355e8dab41f8_isrForCrosstalkSources_1031699_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/57efe522-1e5c-496b-a379-355e8dab41f8_isrForCrosstalkSources_1031699_11.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash 985f0a5342f21cea54914b86b15404d0 parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/cd0654dd-240a-4643-99a2-b7fb705f3b1e_isrForCrosstalkSources_1031709_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/cd0654dd-240a-4643-99a2-b7fb705f3b1e_isrForCrosstalkSources_1031709_14.stderr parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash a1fdc9c46fa9de74bd79eb3be8a19476 parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc46f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/740d1dfb-0c56-4a0f-ba3b-3fdca58c1cd5_isrForCrosstalkSources_1031738_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/740d1dfb-0c56-4a0f-ba3b-3fdca58c1cd5_isrForCrosstalkSources_1031738_39.stderr parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash 1a9bf99bd75ac0bb77be822747126e3e parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc471a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d62877a0-4555-4360-9f17-b90d5c675237_isrForCrosstalkSources_1031714_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d62877a0-4555-4360-9f17-b90d5c675237_isrForCrosstalkSources_1031714_62.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash 43129cc3f8f6676f45b5853207284a20 parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/65cfbfae-0e0d-4c8b-9b3e-a12ede5ea8fc_isrForCrosstalkSources_1031737_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/65cfbfae-0e0d-4c8b-9b3e-a12ede5ea8fc_isrForCrosstalkSources_1031737_6.stderr parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash 976b862b4555f84243d1b15383bc046f parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/285bd959-56e4-4167-9509-76fcc4ed822d_isrForCrosstalkSources_1031723_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/285bd959-56e4-4167-9509-76fcc4ed822d_isrForCrosstalkSources_1031723_29.stderr parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash 0b73e0430d82113af3458883430fa41b parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/944b3532-f330-44ac-9b41-388952d287f0_isrForCrosstalkSources_1031722_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/944b3532-f330-44ac-9b41-388952d287f0_isrForCrosstalkSources_1031722_22.stderr parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1054 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash ccd8a96d8da8a49fbd313c40ecd6dc41 parsl.dataflow.memoization INFO: Task 1054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:09 proc_lsst.multi:170 [INFO] [multi] job_id 17147945 proc_lsst.multi INFO: [multi] job_id 17147945 2024-03-17 04:02:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-17 04:02:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147945 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147945 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17147945 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-17 04:02:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-17 04:02:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-17 04:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 1055 parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d1db1de1-3111-401f-83ba-53765d85cc49_isrForCrosstalkSources_1031727_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d1db1de1-3111-401f-83ba-53765d85cc49_isrForCrosstalkSources_1031727_2.stderr parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1055 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 087bfa58cc789251e48ee0122586923f parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 1056 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5b8b2abd-ce74-44b8-b791-ebc76ac8a79e_isrForCrosstalkSources_1031735_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5b8b2abd-ce74-44b8-b791-ebc76ac8a79e_isrForCrosstalkSources_1031735_58.stderr parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1056 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash 1808d5f7c4894f7bac21656fa3d3e674 parsl.dataflow.memoization INFO: Task 1056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc44040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 1057 parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/62793ff0-afbf-4a72-99c4-6045961e79d0_isrForCrosstalkSources_1031725_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/62793ff0-afbf-4a72-99c4-6045961e79d0_isrForCrosstalkSources_1031725_21.stderr parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1057 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash 7c12a36ac3cd3bd80b3079db251af06b parsl.dataflow.memoization INFO: Task 1057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 1058 parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bdf87c6b-1743-477e-a5a2-0dd693e2f15b_isrForCrosstalkSources_1031734_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bdf87c6b-1743-477e-a5a2-0dd693e2f15b_isrForCrosstalkSources_1031734_18.stderr parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1058 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash 7db72728f90de5835cce0162e9176dc2 parsl.dataflow.memoization INFO: Task 1058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 1059 parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/1a4b66c5-3d99-48a1-bb82-0a6531c5686c_isrForCrosstalkSources_1031718_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/1a4b66c5-3d99-48a1-bb82-0a6531c5686c_isrForCrosstalkSources_1031718_23.stderr parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1059 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash f86752dab14a2b71967f0f2a98ec5ad8 parsl.dataflow.memoization INFO: Task 1059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 1060 parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/c16fff75-686f-4ff5-9103-aa6db593eda4_isrForCrosstalkSources_1031711_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/c16fff75-686f-4ff5-9103-aa6db593eda4_isrForCrosstalkSources_1031711_42.stderr parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1060 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash 5fa8982610c176b0bf13c4d573170db1 parsl.dataflow.memoization INFO: Task 1060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc479c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 1061 parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b5869329-1664-43d8-8531-c58446ca2e98_isrForCrosstalkSources_1031713_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b5869329-1664-43d8-8531-c58446ca2e98_isrForCrosstalkSources_1031713_21.stderr parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1061 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash ea0acda142c8178abd255f494cfc2367 parsl.dataflow.memoization INFO: Task 1061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 1062 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/4fac5e8b-7af2-485f-b38b-9e908effaa79_isrForCrosstalkSources_1031707_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/4fac5e8b-7af2-485f-b38b-9e908effaa79_isrForCrosstalkSources_1031707_45.stderr parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1062 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash 829a3fd992e808962e3187267b94298c parsl.dataflow.memoization INFO: Task 1062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 1063 parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f04c0d2a-dde4-4a20-91f9-1239103e1c4a_isrForCrosstalkSources_1031723_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f04c0d2a-dde4-4a20-91f9-1239103e1c4a_isrForCrosstalkSources_1031723_30.stderr parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1063 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash 3efc665e80e0e179b51c9061d0422fc6 parsl.dataflow.memoization INFO: Task 1063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 1064 parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e71de4b6-b3df-4f2e-b392-e4f82ffa73dd_isrForCrosstalkSources_1031725_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e71de4b6-b3df-4f2e-b392-e4f82ffa73dd_isrForCrosstalkSources_1031725_38.stderr parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1064 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash 5ca17e1540ac6a26aac956512bb48eaa parsl.dataflow.memoization INFO: Task 1064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc477e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 1065 parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/dcb4782d-cf49-422c-ae84-ee5c44ed9bf0_isrForCrosstalkSources_1031721_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/dcb4782d-cf49-422c-ae84-ee5c44ed9bf0_isrForCrosstalkSources_1031721_55.stderr parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1065 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash 1649e96f95eefce928c30145fc175711 parsl.dataflow.memoization INFO: Task 1065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 1066 parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/07f8b70c-4796-4b3f-b930-1b0e57edd38f_isrForCrosstalkSources_1031728_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/07f8b70c-4796-4b3f-b930-1b0e57edd38f_isrForCrosstalkSources_1031728_58.stderr parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1066 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash 19f258a4698482ca2451089f7e13aec9 parsl.dataflow.memoization INFO: Task 1066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc476a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 1067 parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/32eb516c-4702-40f6-af98-ea318518a5d1_isrForCrosstalkSources_1031701_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/32eb516c-4702-40f6-af98-ea318518a5d1_isrForCrosstalkSources_1031701_39.stderr parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1067 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash 3e17dda319bf797b23cea267522505c8 parsl.dataflow.memoization INFO: Task 1067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc474c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 1068 parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fb2d0e30-8b28-4b11-9454-df57545534cb_isrForCrosstalkSources_1031732_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fb2d0e30-8b28-4b11-9454-df57545534cb_isrForCrosstalkSources_1031732_12.stderr parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1068 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash 9c574277e4f1120c7e4b8d3a1263c2dc parsl.dataflow.memoization INFO: Task 1068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 1069 parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/0a8445a2-468f-4bab-a2cb-1201fe593041_isrForCrosstalkSources_1031716_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/0a8445a2-468f-4bab-a2cb-1201fe593041_isrForCrosstalkSources_1031716_15.stderr parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1069 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash 9a49dbde10f9a415fca09967d4ed9331 parsl.dataflow.memoization INFO: Task 1069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 1070 parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/dea7c380-4dd8-489a-aec6-e2a9889fe897_isrForCrosstalkSources_1031707_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/dea7c380-4dd8-489a-aec6-e2a9889fe897_isrForCrosstalkSources_1031707_25.stderr parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1070 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 2fc61610b67c9a8bf69b93bac0199b70 parsl.dataflow.memoization INFO: Task 1070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 1071 parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/b57a6446-2a82-4a69-9d9b-fd9e7803fa2b_isrForCrosstalkSources_1031735_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/b57a6446-2a82-4a69-9d9b-fd9e7803fa2b_isrForCrosstalkSources_1031735_36.stderr parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1071 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash 5a7d95e0261d776387dae890de019c71 parsl.dataflow.memoization INFO: Task 1071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc472e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 1072 parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4f18ac1c-6e4c-4fa2-b219-4ca9a058b07f_isrForCrosstalkSources_1031737_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4f18ac1c-6e4c-4fa2-b219-4ca9a058b07f_isrForCrosstalkSources_1031737_29.stderr parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1072 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 70fdf952183fb3b8fe541890071bf983 parsl.dataflow.memoization INFO: Task 1072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 1073 parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/817d4905-a3be-4495-ab1d-0749f498e8dd_isrForCrosstalkSources_1031725_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/817d4905-a3be-4495-ab1d-0749f498e8dd_isrForCrosstalkSources_1031725_31.stderr parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1073 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash 46750b0a24ec7e6b0a4fbe300efbc8c5 parsl.dataflow.memoization INFO: Task 1073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 1074 parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/3b72fde3-6805-486f-9f02-a9dbb9cd2a28_isrForCrosstalkSources_1031708_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/3b72fde3-6805-486f-9f02-a9dbb9cd2a28_isrForCrosstalkSources_1031708_55.stderr parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1074 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash 22ebbf1653230a9b69b21c4a14e0b987 parsl.dataflow.memoization INFO: Task 1074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 1075 parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e14213dd-9ea6-4285-bb66-e8a18cc4f94e_isrForCrosstalkSources_1031738_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e14213dd-9ea6-4285-bb66-e8a18cc4f94e_isrForCrosstalkSources_1031738_9.stderr parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1075 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash e747f77f80b708d31c15233eb9635e49 parsl.dataflow.memoization INFO: Task 1075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 1076 parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/734ba3f2-e129-44c5-9b4d-60674cfae774_isrForCrosstalkSources_1031707_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/734ba3f2-e129-44c5-9b4d-60674cfae774_isrForCrosstalkSources_1031707_30.stderr parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1076 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash 06449b31be24e7dc498dac4c8a9d0589 parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1077 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0992e56e-1be0-49ad-9087-2283f8396051_isrForCrosstalkSources_1031737_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0992e56e-1be0-49ad-9087-2283f8396051_isrForCrosstalkSources_1031737_44.stderr parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1077 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash 80e1dff7b873a31cff5f051484e38333 parsl.dataflow.memoization INFO: Task 1077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 1078 parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a51c86d9-21ae-48e0-8c7e-bb01cbd38692_isrForCrosstalkSources_1031701_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a51c86d9-21ae-48e0-8c7e-bb01cbd38692_isrForCrosstalkSources_1031701_12.stderr parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1078 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash 2c6dfef37ab3c4607382115cfd75ff8a parsl.dataflow.memoization INFO: Task 1078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 1079 parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e6bee9d1-efb4-4399-b7d4-464ff7a4d0b9_isrForCrosstalkSources_1031716_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e6bee9d1-efb4-4399-b7d4-464ff7a4d0b9_isrForCrosstalkSources_1031716_55.stderr parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1079 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 25f22b95ba660dfb30d5e14b37601654 parsl.dataflow.memoization INFO: Task 1079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 1080 parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/eaf72f9d-9dc1-473b-8dee-00b63a29d3a9_isrForCrosstalkSources_1031700_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/eaf72f9d-9dc1-473b-8dee-00b63a29d3a9_isrForCrosstalkSources_1031700_21.stderr parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1080 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash f0eb5625eb1cdee1124e43b8d08b99f7 parsl.dataflow.memoization INFO: Task 1080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 1081 parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/653ca142-6ee9-4e19-8e63-6dbdab2f7cde_isrForCrosstalkSources_1031712_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/653ca142-6ee9-4e19-8e63-6dbdab2f7cde_isrForCrosstalkSources_1031712_48.stderr parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1081 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash d84c30d878ac0e031c5b6e2e38fc5ce2 parsl.dataflow.memoization INFO: Task 1081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 1082 parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7b9bfda0-3b05-4d42-b985-e682c94da391_isrForCrosstalkSources_1031725_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7b9bfda0-3b05-4d42-b985-e682c94da391_isrForCrosstalkSources_1031725_19.stderr parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1082 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash f27902c1d41a0154fcf6f52b0ecea14c parsl.dataflow.memoization INFO: Task 1082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 1083 parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/25cf3cf5-5cd1-4639-ab00-1f6131841bef_isrForCrosstalkSources_1031713_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/25cf3cf5-5cd1-4639-ab00-1f6131841bef_isrForCrosstalkSources_1031713_33.stderr parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1083 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 5427a1d4f1687f2fbb60918b74750e62 parsl.dataflow.memoization INFO: Task 1083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 1084 parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d2017012-fe24-43b8-bc78-8f9fb7df6fcd_isrForCrosstalkSources_1031722_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d2017012-fe24-43b8-bc78-8f9fb7df6fcd_isrForCrosstalkSources_1031722_6.stderr parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1084 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash 16ddabc2c1356185e87466408a65e2bd parsl.dataflow.memoization INFO: Task 1084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 1085 parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/da0fa9aa-f5e5-4c80-ae81-26771e3b4992_isrForCrosstalkSources_1031727_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/da0fa9aa-f5e5-4c80-ae81-26771e3b4992_isrForCrosstalkSources_1031727_27.stderr parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1085 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash c9493fd291b85882badf6ed9db3ab173 parsl.dataflow.memoization INFO: Task 1085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 1086 parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8fa0d8ea-f63e-453e-b856-c119e17243b4_isrForCrosstalkSources_1031700_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8fa0d8ea-f63e-453e-b856-c119e17243b4_isrForCrosstalkSources_1031700_28.stderr parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1086 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash 9ab137d9ba07f002dc67f28ab3dcc368 parsl.dataflow.memoization INFO: Task 1086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 1087 parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/792a809a-f524-4732-b83c-df55aa05541f_isrForCrosstalkSources_1031734_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/792a809a-f524-4732-b83c-df55aa05541f_isrForCrosstalkSources_1031734_60.stderr parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1087 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash 51817f488111e057a812b1ab5b7cd69e parsl.dataflow.memoization INFO: Task 1087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 1088 parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b97a0cb1-e67f-4691-8e5a-c8ec4584f919_isrForCrosstalkSources_1031713_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b97a0cb1-e67f-4691-8e5a-c8ec4584f919_isrForCrosstalkSources_1031713_30.stderr parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1088 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash 7592b5a8f5ce2cd15b13addb9d522e06 parsl.dataflow.memoization INFO: Task 1088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 1089 parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/796e6d26-8357-448e-813a-2fba6a5a6947_isrForCrosstalkSources_1031723_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/796e6d26-8357-448e-813a-2fba6a5a6947_isrForCrosstalkSources_1031723_20.stderr parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1089 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash 1212a57bfb49345327c60640bca2e33c parsl.dataflow.memoization INFO: Task 1089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 1090 parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/ab4fa0e9-4040-4dda-ab47-cc05b129fede_isrForCrosstalkSources_1031707_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/ab4fa0e9-4040-4dda-ab47-cc05b129fede_isrForCrosstalkSources_1031707_10.stderr parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1090 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash 3d83ab934bb79a58b550ed17f45c4508 parsl.dataflow.memoization INFO: Task 1090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 1091 parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/0622e87d-9068-4954-b99b-e83f41d52337_isrForCrosstalkSources_1031723_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/0622e87d-9068-4954-b99b-e83f41d52337_isrForCrosstalkSources_1031723_37.stderr parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1091 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash bc4880477ad47d323ccb083f432649e0 parsl.dataflow.memoization INFO: Task 1091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 1092 parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0f1dbb2c-c1b3-4da8-8055-7d12e4b6c09c_isrForCrosstalkSources_1031717_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0f1dbb2c-c1b3-4da8-8055-7d12e4b6c09c_isrForCrosstalkSources_1031717_16.stderr parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1092 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash caf122574eb0a4e6f863b0fb6f9a1cff parsl.dataflow.memoization INFO: Task 1092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 1093 parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/4900ac4e-eea4-4f22-baf8-34e37503d4c5_isrForCrosstalkSources_1031713_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/4900ac4e-eea4-4f22-baf8-34e37503d4c5_isrForCrosstalkSources_1031713_37.stderr parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1093 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash 40ab67faf5a1d14620ea546b6dd9d2aa parsl.dataflow.memoization INFO: Task 1093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 1094 parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2c939247-974f-4d5b-9d27-8b340f5ca313_isrForCrosstalkSources_1031724_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2c939247-974f-4d5b-9d27-8b340f5ca313_isrForCrosstalkSources_1031724_43.stderr parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1094 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 371372799656ff50457ce9677700e882 parsl.dataflow.memoization INFO: Task 1094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 1095 parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/0e8f1d21-584b-4ff2-b2be-991832986e63_isrForCrosstalkSources_1031702_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/0e8f1d21-584b-4ff2-b2be-991832986e63_isrForCrosstalkSources_1031702_46.stderr parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1095 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash 957b986e909d9db07293b10b0361c9b4 parsl.dataflow.memoization INFO: Task 1095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 1096 parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5ef68d23-93f1-418c-93ae-8ca0313a8182_isrForCrosstalkSources_1031736_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5ef68d23-93f1-418c-93ae-8ca0313a8182_isrForCrosstalkSources_1031736_16.stderr parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1096 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash 55e3f57d94e490434ef34dbe1198d09c parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 1097 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e3749196-8131-4148-8c00-bfd9c07b0382_isrForCrosstalkSources_1031717_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e3749196-8131-4148-8c00-bfd9c07b0382_isrForCrosstalkSources_1031717_57.stderr parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1097 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash ea254b5f5105e0ec05bce04808007821 parsl.dataflow.memoization INFO: Task 1097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 1098 parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/0e068ec8-319c-49b8-9792-844f6a1ca038_isrForCrosstalkSources_1031738_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/0e068ec8-319c-49b8-9792-844f6a1ca038_isrForCrosstalkSources_1031738_52.stderr parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1098 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash 02a185937f592db73f24484aeaceb25e parsl.dataflow.memoization INFO: Task 1098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 1099 parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/210a3a76-f299-4b2e-8d53-b654da98d484_isrForCrosstalkSources_1031700_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/210a3a76-f299-4b2e-8d53-b654da98d484_isrForCrosstalkSources_1031700_23.stderr parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1099 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash e13c53a1e2ef465f1c1c9fac72d62bb7 parsl.dataflow.memoization INFO: Task 1099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 1100 parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/b738912f-9b29-4488-b798-dbeb52c0e98d_isrForCrosstalkSources_1031733_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/b738912f-9b29-4488-b798-dbeb52c0e98d_isrForCrosstalkSources_1031733_30.stderr parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash 597d3ade1430261ecfb74b7c93cab6b5 parsl.dataflow.memoization INFO: Task 1100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 1101 parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f87db46b-039e-4e06-a667-32eed3c5fd91_isrForCrosstalkSources_1031732_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f87db46b-039e-4e06-a667-32eed3c5fd91_isrForCrosstalkSources_1031732_8.stderr parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 2cfd9efe689eeb2e8cf694f6cb459223 parsl.dataflow.memoization INFO: Task 1101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 1102 parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/88af585e-1239-49dc-aa84-1a5c3c6d9efd_isrForCrosstalkSources_1031729_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/88af585e-1239-49dc-aa84-1a5c3c6d9efd_isrForCrosstalkSources_1031729_47.stderr parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash 271bfb33f8bbe5913d5bd1682ae692fa parsl.dataflow.memoization INFO: Task 1102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 1103 parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fb248c28-76b6-4ebe-8eb3-fbbe800877da_isrForCrosstalkSources_1031707_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fb248c28-76b6-4ebe-8eb3-fbbe800877da_isrForCrosstalkSources_1031707_12.stderr parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash 5551e1df58c119627a80f88139bbfd33 parsl.dataflow.memoization INFO: Task 1103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 1104 parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8f3d0ea5-775a-438a-a609-435951c7b195_isrForCrosstalkSources_1031714_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8f3d0ea5-775a-438a-a609-435951c7b195_isrForCrosstalkSources_1031714_55.stderr parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash 6e444ffd2aaba9a11bd186eba731f73c parsl.dataflow.memoization INFO: Task 1104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 1105 parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ecc7a798-e303-4933-9c6a-cb7694d7a437_isrForCrosstalkSources_1031738_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ecc7a798-e303-4933-9c6a-cb7694d7a437_isrForCrosstalkSources_1031738_32.stderr parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 08fbd6b8ecda056b53c74a38fb152847 parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 1106 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/d23d2b20-c6f5-4d68-9189-e9fdad16e29b_isrForCrosstalkSources_1031732_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/d23d2b20-c6f5-4d68-9189-e9fdad16e29b_isrForCrosstalkSources_1031732_44.stderr parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash 3d5c95398d4f7a9c5c4b16f55dce74b6 parsl.dataflow.memoization INFO: Task 1106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 1107 parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7c3283f2-05c7-4d18-8b5b-35521d71053c_isrForCrosstalkSources_1031725_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7c3283f2-05c7-4d18-8b5b-35521d71053c_isrForCrosstalkSources_1031725_5.stderr parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash 3e2b11a290fc5e01c1eea79b43f717f1 parsl.dataflow.memoization INFO: Task 1107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 1108 parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/baf602bf-5009-47ff-bf79-3bdc9c100df5_isrForCrosstalkSources_1031700_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/baf602bf-5009-47ff-bf79-3bdc9c100df5_isrForCrosstalkSources_1031700_41.stderr parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash 0674b049706b4fccea97fd67855d82f0 parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 1109 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e1dd2b03-e249-448b-b446-c3fd14c8c98c_isrForCrosstalkSources_1031731_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e1dd2b03-e249-448b-b446-c3fd14c8c98c_isrForCrosstalkSources_1031731_13.stderr parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash f7a14540c9836a5bd9055d61a2f49e5a parsl.dataflow.memoization INFO: Task 1109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 1110 parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/213e916a-5542-4dfc-b003-5eca744dfe3f_isrForCrosstalkSources_1031736_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/213e916a-5542-4dfc-b003-5eca744dfe3f_isrForCrosstalkSources_1031736_21.stderr parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash 71e1086c116538e80497504324e62541 parsl.dataflow.memoization INFO: Task 1110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 1111 parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b5056f8f-1747-4fe2-92c6-d126c05feeec_isrForCrosstalkSources_1031714_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b5056f8f-1747-4fe2-92c6-d126c05feeec_isrForCrosstalkSources_1031714_44.stderr parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash 904729a2321a79947ec2bcfe0e5238c5 parsl.dataflow.memoization INFO: Task 1111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 1112 parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/b3385e8a-723d-49f2-b952-daa935765e75_isrForCrosstalkSources_1031719_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/b3385e8a-723d-49f2-b952-daa935765e75_isrForCrosstalkSources_1031719_15.stderr parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash b318856f29674fddbf28fa812b67dc3b parsl.dataflow.memoization INFO: Task 1112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 1113 parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/99e4f978-ee47-442d-b0c4-2dcb40750ca4_isrForCrosstalkSources_1031730_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/99e4f978-ee47-442d-b0c4-2dcb40750ca4_isrForCrosstalkSources_1031730_11.stderr parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash 56942ac53e23d148953b145a14ac9f36 parsl.dataflow.memoization INFO: Task 1113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 1114 parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6da7c709-f61f-4a34-92ae-0deac999dc79_isrForCrosstalkSources_1031721_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6da7c709-f61f-4a34-92ae-0deac999dc79_isrForCrosstalkSources_1031721_10.stderr parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash 9107ded8cf2cb24cc6a088ccd4fdcd5a parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 1115 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bab6070c-be9d-4fab-8f1c-0cb9df3330af_isrForCrosstalkSources_1031734_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bab6070c-be9d-4fab-8f1c-0cb9df3330af_isrForCrosstalkSources_1031734_32.stderr parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash cf7d9b57f00f6abb07effd174317eb87 parsl.dataflow.memoization INFO: Task 1115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 1116 parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a4cee620-f244-4344-bc79-1246507fd3ee_isrForCrosstalkSources_1031730_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a4cee620-f244-4344-bc79-1246507fd3ee_isrForCrosstalkSources_1031730_13.stderr parsl.dataflow.dflow DEBUG: Task 1116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1116 has memoization hash d03010790b8030b64840e22d84f6ac82 parsl.dataflow.memoization INFO: Task 1116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1116 try 0 launched on executor multi with executor id 1117 parsl.dataflow.dflow INFO: Standard output for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a7c8ac75-b9d9-4abf-a9e4-944433eb7817_isrForCrosstalkSources_1031726_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a7c8ac75-b9d9-4abf-a9e4-944433eb7817_isrForCrosstalkSources_1031726_60.stderr parsl.dataflow.dflow DEBUG: Task 1117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1117 has memoization hash 1deafd186cceb0a61eccac84752644ba parsl.dataflow.memoization INFO: Task 1117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1117 try 0 launched on executor multi with executor id 1118 parsl.dataflow.dflow INFO: Standard output for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/af3f3e77-1d4f-41d5-8491-92ae6c597f68_isrForCrosstalkSources_1031710_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/af3f3e77-1d4f-41d5-8491-92ae6c597f68_isrForCrosstalkSources_1031710_60.stderr parsl.dataflow.dflow DEBUG: Task 1118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1118 has memoization hash 64f7145985a84d4174b66c169715c489 parsl.dataflow.memoization INFO: Task 1118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1118 try 0 launched on executor multi with executor id 1119 parsl.dataflow.dflow INFO: Standard output for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1c062baa-c013-443a-a094-f7937cc14c64_isrForCrosstalkSources_1031701_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1c062baa-c013-443a-a094-f7937cc14c64_isrForCrosstalkSources_1031701_17.stderr parsl.dataflow.dflow DEBUG: Task 1119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1119 has memoization hash 504263e12f875570a047e3531db9b9b0 parsl.dataflow.memoization INFO: Task 1119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1119 try 0 launched on executor multi with executor id 1120 parsl.dataflow.dflow INFO: Standard output for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/aa50dcbd-b0eb-4d97-be20-b744e5fc415e_isrForCrosstalkSources_1031738_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/aa50dcbd-b0eb-4d97-be20-b744e5fc415e_isrForCrosstalkSources_1031738_10.stderr parsl.dataflow.dflow DEBUG: Task 1120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1120 has memoization hash 47ba661caad84e76f24607322f1ec0bc parsl.dataflow.memoization INFO: Task 1120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1120 try 0 launched on executor multi with executor id 1121 parsl.dataflow.dflow INFO: Standard output for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/d1925aa7-b77e-4ab6-8265-8978f123b868_isrForCrosstalkSources_1031702_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/d1925aa7-b77e-4ab6-8265-8978f123b868_isrForCrosstalkSources_1031702_34.stderr parsl.dataflow.dflow DEBUG: Task 1121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1121 has memoization hash 931bd8acb22109eb98fcc20c236d93d8 parsl.dataflow.memoization INFO: Task 1121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1121 try 0 launched on executor multi with executor id 1122 parsl.dataflow.dflow INFO: Standard output for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/0a09788c-7e78-4dba-8e73-e0a8c646a082_isrForCrosstalkSources_1031715_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/0a09788c-7e78-4dba-8e73-e0a8c646a082_isrForCrosstalkSources_1031715_10.stderr parsl.dataflow.dflow DEBUG: Task 1122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1122 has memoization hash f6395e67214033595ac82da1daf74931 parsl.dataflow.memoization INFO: Task 1122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1122 try 0 launched on executor multi with executor id 1123 parsl.dataflow.dflow INFO: Standard output for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/6b220235-8001-4299-b9c5-bb6f706eb970_isrForCrosstalkSources_1031714_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/6b220235-8001-4299-b9c5-bb6f706eb970_isrForCrosstalkSources_1031714_59.stderr parsl.dataflow.dflow DEBUG: Task 1123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1123 has memoization hash 9dff0340b0055ca95ec28d635e535c7f parsl.dataflow.memoization INFO: Task 1123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1123 try 0 launched on executor multi with executor id 1124 parsl.dataflow.dflow INFO: Standard output for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/16e89f2d-ceee-4337-baf4-5fa085763497_isrForCrosstalkSources_1031737_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/16e89f2d-ceee-4337-baf4-5fa085763497_isrForCrosstalkSources_1031737_1.stderr parsl.dataflow.dflow DEBUG: Task 1124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1124 has memoization hash 1679691fcca417189d15412d7fdfc041 parsl.dataflow.memoization INFO: Task 1124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1124 try 0 launched on executor multi with executor id 1125 parsl.dataflow.dflow INFO: Standard output for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0589e46d-f053-4acf-958d-d9cf151497dc_isrForCrosstalkSources_1031737_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0589e46d-f053-4acf-958d-d9cf151497dc_isrForCrosstalkSources_1031737_57.stderr parsl.dataflow.dflow DEBUG: Task 1125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1125 has memoization hash 99a360fe7b186328d7c3cd84553f5b80 parsl.dataflow.memoization INFO: Task 1125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1125 try 0 launched on executor multi with executor id 1126 parsl.dataflow.dflow INFO: Standard output for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c00fcf87-d245-4132-b739-9d5a0dd6c497_isrForCrosstalkSources_1031702_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c00fcf87-d245-4132-b739-9d5a0dd6c497_isrForCrosstalkSources_1031702_7.stderr parsl.dataflow.dflow DEBUG: Task 1126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1126 has memoization hash 8793531ee81e2288f9a4a1925b77608c parsl.dataflow.memoization INFO: Task 1126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1126 try 0 launched on executor multi with executor id 1127 parsl.dataflow.dflow INFO: Standard output for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/813dab32-9ded-4233-baac-18792ac33ca8_isrForCrosstalkSources_1031704_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/813dab32-9ded-4233-baac-18792ac33ca8_isrForCrosstalkSources_1031704_7.stderr parsl.dataflow.dflow DEBUG: Task 1127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1127 has memoization hash c6d37489ce887420de79ca681792403b parsl.dataflow.memoization INFO: Task 1127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1127 try 0 launched on executor multi with executor id 1128 parsl.dataflow.dflow INFO: Standard output for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bde66494-f855-47b4-bd48-97405980f094_isrForCrosstalkSources_1031734_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bde66494-f855-47b4-bd48-97405980f094_isrForCrosstalkSources_1031734_51.stderr parsl.dataflow.dflow DEBUG: Task 1128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1128 has memoization hash 7997808485b5ab0dc1ae4c26949e47c0 parsl.dataflow.memoization INFO: Task 1128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1128 try 0 launched on executor multi with executor id 1129 parsl.dataflow.dflow INFO: Standard output for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/09626f37-a8e4-4b54-9a1d-11cb036473c4_isrForCrosstalkSources_1031724_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/09626f37-a8e4-4b54-9a1d-11cb036473c4_isrForCrosstalkSources_1031724_9.stderr parsl.dataflow.dflow DEBUG: Task 1129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1129 has memoization hash 4a0e32fc7103482b6e2d6b3a1696a751 parsl.dataflow.memoization INFO: Task 1129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1129 try 0 launched on executor multi with executor id 1130 parsl.dataflow.dflow INFO: Standard output for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/9a21e67d-8967-4ea1-8909-b224f95bd939_isrForCrosstalkSources_1031704_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/9a21e67d-8967-4ea1-8909-b224f95bd939_isrForCrosstalkSources_1031704_57.stderr parsl.dataflow.dflow DEBUG: Task 1130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1130 has memoization hash 66934fcfc7a8d1cf9c91097e294fc89b parsl.dataflow.memoization INFO: Task 1130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:10 proc_lsst.multi:170 [INFO] [multi] job_id 17147946 proc_lsst.multi INFO: [multi] job_id 17147946 2024-03-17 04:02:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-17 04:02:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147946 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147946 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17147946 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-17 04:02:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-17 04:02:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-17 04:02:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1130 try 0 launched on executor multi with executor id 1131 parsl.dataflow.dflow INFO: Standard output for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e24169f6-5741-4941-b1a5-d1d373b4c42d_isrForCrosstalkSources_1031730_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e24169f6-5741-4941-b1a5-d1d373b4c42d_isrForCrosstalkSources_1031730_58.stderr parsl.dataflow.dflow DEBUG: Task 1131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1131 has memoization hash 077164c47713c9e8276821cd5467d67a parsl.dataflow.memoization INFO: Task 1131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fd872e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1131 try 0 launched on executor multi with executor id 1132 parsl.dataflow.dflow INFO: Standard output for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/507214e5-74fc-40c5-8ab5-4c6e498cc836_isrForCrosstalkSources_1031727_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/507214e5-74fc-40c5-8ab5-4c6e498cc836_isrForCrosstalkSources_1031727_3.stderr parsl.dataflow.dflow DEBUG: Task 1132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1132 has memoization hash 25acf4d1355bf2760fe01e3eede202e4 parsl.dataflow.memoization INFO: Task 1132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e8ebe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1132 try 0 launched on executor multi with executor id 1133 parsl.dataflow.dflow INFO: Standard output for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a4643fa5-b0a2-4596-8d78-2bcb748b819b_isrForCrosstalkSources_1031730_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a4643fa5-b0a2-4596-8d78-2bcb748b819b_isrForCrosstalkSources_1031730_61.stderr parsl.dataflow.dflow DEBUG: Task 1133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1133 has memoization hash 9866028d501a4ea0812083eb87814079 parsl.dataflow.memoization INFO: Task 1133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1133 try 0 launched on executor multi with executor id 1134 parsl.dataflow.dflow INFO: Standard output for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7b6ff05e-f455-449c-853e-ad5807a8a977_isrForCrosstalkSources_1031729_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7b6ff05e-f455-449c-853e-ad5807a8a977_isrForCrosstalkSources_1031729_53.stderr parsl.dataflow.dflow DEBUG: Task 1134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1134 has memoization hash bc10bcc00827a5e455008a60c345a808 parsl.dataflow.memoization INFO: Task 1134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1134 try 0 launched on executor multi with executor id 1135 parsl.dataflow.dflow INFO: Standard output for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/124402a8-ec66-468c-9983-4ab26cbaba12_isrForCrosstalkSources_1031733_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/124402a8-ec66-468c-9983-4ab26cbaba12_isrForCrosstalkSources_1031733_6.stderr parsl.dataflow.dflow DEBUG: Task 1135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1135 has memoization hash 8b4a7a0e84996e22af263f1fe8185150 parsl.dataflow.memoization INFO: Task 1135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1135 try 0 launched on executor multi with executor id 1136 parsl.dataflow.dflow INFO: Standard output for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/604d9c5f-a0d1-4543-abc9-77828dc0c449_isrForCrosstalkSources_1031704_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/604d9c5f-a0d1-4543-abc9-77828dc0c449_isrForCrosstalkSources_1031704_28.stderr parsl.dataflow.dflow DEBUG: Task 1136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1136 has memoization hash 4a62d7946bf2f2874df2e77cde98123b parsl.dataflow.memoization INFO: Task 1136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1136 try 0 launched on executor multi with executor id 1137 parsl.dataflow.dflow INFO: Standard output for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/aeb2a75c-4943-4e5a-be9e-96be6e11bb80_isrForCrosstalkSources_1031723_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/aeb2a75c-4943-4e5a-be9e-96be6e11bb80_isrForCrosstalkSources_1031723_5.stderr parsl.dataflow.dflow DEBUG: Task 1137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1137 has memoization hash 46fcd55a4b8b8b275760fdb6c4e2fcfe parsl.dataflow.memoization INFO: Task 1137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1137 try 0 launched on executor multi with executor id 1138 parsl.dataflow.dflow INFO: Standard output for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/899b9a50-c6f8-4aa2-a6cb-d41bc59ab94c_isrForCrosstalkSources_1031707_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/899b9a50-c6f8-4aa2-a6cb-d41bc59ab94c_isrForCrosstalkSources_1031707_62.stderr parsl.dataflow.dflow DEBUG: Task 1138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1138 has memoization hash a723e92c4968860843735529a8803508 parsl.dataflow.memoization INFO: Task 1138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1138 try 0 launched on executor multi with executor id 1139 parsl.dataflow.dflow INFO: Standard output for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/917aa082-ef62-4101-a4e6-4e40a353a654_isrForCrosstalkSources_1031711_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/917aa082-ef62-4101-a4e6-4e40a353a654_isrForCrosstalkSources_1031711_5.stderr parsl.dataflow.dflow DEBUG: Task 1139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1139 has memoization hash 5c62031852442535cf492dc312fb722b parsl.dataflow.memoization INFO: Task 1139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1139 try 0 launched on executor multi with executor id 1140 parsl.dataflow.dflow INFO: Standard output for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c024fb2a-e6bd-472a-a14a-d8228ac41501_isrForCrosstalkSources_1031726_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c024fb2a-e6bd-472a-a14a-d8228ac41501_isrForCrosstalkSources_1031726_61.stderr parsl.dataflow.dflow DEBUG: Task 1140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1140 has memoization hash f4fbee516e0d8d597986fc9dc1f8a1a1 parsl.dataflow.memoization INFO: Task 1140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1140 try 0 launched on executor multi with executor id 1141 parsl.dataflow.dflow INFO: Standard output for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5f68dcc8-a0e5-43e5-b4fa-51819cfd4f44_isrForCrosstalkSources_1031730_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5f68dcc8-a0e5-43e5-b4fa-51819cfd4f44_isrForCrosstalkSources_1031730_15.stderr parsl.dataflow.dflow DEBUG: Task 1141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1141 has memoization hash 4a313b87ce5488f86259f39ce22fdf91 parsl.dataflow.memoization INFO: Task 1141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1141 try 0 launched on executor multi with executor id 1142 parsl.dataflow.dflow INFO: Standard output for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/bd72e86e-f4d3-4036-909a-a58075eeccaa_isrForCrosstalkSources_1031700_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/bd72e86e-f4d3-4036-909a-a58075eeccaa_isrForCrosstalkSources_1031700_49.stderr parsl.dataflow.dflow DEBUG: Task 1142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1142 has memoization hash 79dcec1b8d1e6c09b660660099744f31 parsl.dataflow.memoization INFO: Task 1142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1142 try 0 launched on executor multi with executor id 1143 parsl.dataflow.dflow INFO: Standard output for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/2d1ff028-346c-4995-b1f9-835e040121da_isrForCrosstalkSources_1031717_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/2d1ff028-346c-4995-b1f9-835e040121da_isrForCrosstalkSources_1031717_55.stderr parsl.dataflow.dflow DEBUG: Task 1143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1143 has memoization hash 61242f41acbaf9dde5cf380bd0a29322 parsl.dataflow.memoization INFO: Task 1143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1143 try 0 launched on executor multi with executor id 1144 parsl.dataflow.dflow INFO: Standard output for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/b7358991-784a-4474-bc7c-b79db9c30ac0_isrForCrosstalkSources_1031706_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/b7358991-784a-4474-bc7c-b79db9c30ac0_isrForCrosstalkSources_1031706_43.stderr parsl.dataflow.dflow DEBUG: Task 1144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1144 has memoization hash 0d807b351ea74a522ba776338823b421 parsl.dataflow.memoization INFO: Task 1144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1144 try 0 launched on executor multi with executor id 1145 parsl.dataflow.dflow INFO: Standard output for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/04ca0c34-8666-4cb9-87ea-c2fb4453a8c6_isrForCrosstalkSources_1031731_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/04ca0c34-8666-4cb9-87ea-c2fb4453a8c6_isrForCrosstalkSources_1031731_7.stderr parsl.dataflow.dflow DEBUG: Task 1145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1145 has memoization hash 7eb04c2ce12f39e9f6f018701bfee348 parsl.dataflow.memoization INFO: Task 1145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1145 try 0 launched on executor multi with executor id 1146 parsl.dataflow.dflow INFO: Standard output for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c5fcdb55-8f39-45da-9e48-37c0f8beff8e_isrForCrosstalkSources_1031708_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c5fcdb55-8f39-45da-9e48-37c0f8beff8e_isrForCrosstalkSources_1031708_1.stderr parsl.dataflow.dflow DEBUG: Task 1146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1146 has memoization hash 5ed0495d849d959979aad98f97cb26c3 parsl.dataflow.memoization INFO: Task 1146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1146 try 0 launched on executor multi with executor id 1147 parsl.dataflow.dflow INFO: Standard output for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3514c62a-05d7-4983-9be3-4890c8db826a_isrForCrosstalkSources_1031716_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3514c62a-05d7-4983-9be3-4890c8db826a_isrForCrosstalkSources_1031716_26.stderr parsl.dataflow.dflow DEBUG: Task 1147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1147 has memoization hash ca9bf9101e3eab7e32449ba56dea9e8b parsl.dataflow.memoization INFO: Task 1147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1147 try 0 launched on executor multi with executor id 1148 parsl.dataflow.dflow INFO: Standard output for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a5ebbb34-445a-4e58-8642-236c94e1dba7_isrForCrosstalkSources_1031713_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a5ebbb34-445a-4e58-8642-236c94e1dba7_isrForCrosstalkSources_1031713_43.stderr parsl.dataflow.dflow DEBUG: Task 1148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1148 has memoization hash e6e00ee77488356ba92178c686b22756 parsl.dataflow.memoization INFO: Task 1148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1148 try 0 launched on executor multi with executor id 1149 parsl.dataflow.dflow INFO: Standard output for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b5161d18-796d-4660-8f0e-f0180c491b69_isrForCrosstalkSources_1031726_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b5161d18-796d-4660-8f0e-f0180c491b69_isrForCrosstalkSources_1031726_43.stderr parsl.dataflow.dflow DEBUG: Task 1149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1149 has memoization hash fa87eb18a7500909d62081ec5deb5131 parsl.dataflow.memoization INFO: Task 1149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1149 try 0 launched on executor multi with executor id 1150 parsl.dataflow.dflow INFO: Standard output for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/de783515-272a-4070-9cd3-182f5e598ae3_isrForCrosstalkSources_1031733_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/de783515-272a-4070-9cd3-182f5e598ae3_isrForCrosstalkSources_1031733_22.stderr parsl.dataflow.dflow DEBUG: Task 1150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1150 has memoization hash 8fd4c665a7b97d5e553d28c2a093c72d parsl.dataflow.memoization INFO: Task 1150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1150 try 0 launched on executor multi with executor id 1151 parsl.dataflow.dflow INFO: Standard output for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1cf320bc-6277-4352-8ac3-8c199495f408_isrForCrosstalkSources_1031701_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1cf320bc-6277-4352-8ac3-8c199495f408_isrForCrosstalkSources_1031701_51.stderr parsl.dataflow.dflow DEBUG: Task 1151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1151 has memoization hash 4a8b734e4cfb2cfba43ef8bd3b7ecf73 parsl.dataflow.memoization INFO: Task 1151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1151 try 0 launched on executor multi with executor id 1152 parsl.dataflow.dflow INFO: Standard output for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c88dc2bf-eea2-410f-ab34-051c45d265f5_isrForCrosstalkSources_1031716_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c88dc2bf-eea2-410f-ab34-051c45d265f5_isrForCrosstalkSources_1031716_25.stderr parsl.dataflow.dflow DEBUG: Task 1152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1152 has memoization hash e73abd06d3cb78b2d3b9077cb298180b parsl.dataflow.memoization INFO: Task 1152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1152 try 0 launched on executor multi with executor id 1153 parsl.dataflow.dflow INFO: Standard output for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/b409c3ea-d581-478a-8bdd-e0acfec25225_isrForCrosstalkSources_1031717_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/b409c3ea-d581-478a-8bdd-e0acfec25225_isrForCrosstalkSources_1031717_29.stderr parsl.dataflow.dflow DEBUG: Task 1153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1153 has memoization hash fc25012eb16b693d4a226047e2397e38 parsl.dataflow.memoization INFO: Task 1153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1153 try 0 launched on executor multi with executor id 1154 parsl.dataflow.dflow INFO: Standard output for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/3e1a216d-d118-4f5d-8f2e-cc8a14b453ac_isrForCrosstalkSources_1031701_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/3e1a216d-d118-4f5d-8f2e-cc8a14b453ac_isrForCrosstalkSources_1031701_11.stderr parsl.dataflow.dflow DEBUG: Task 1154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1154 has memoization hash 2fb79c61aa9a8215e8c744ac7aabf9be parsl.dataflow.memoization INFO: Task 1154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1154 try 0 launched on executor multi with executor id 1155 parsl.dataflow.dflow INFO: Standard output for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/cc8e580e-e698-4282-9575-29eee14aba62_isrForCrosstalkSources_1031735_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/cc8e580e-e698-4282-9575-29eee14aba62_isrForCrosstalkSources_1031735_7.stderr parsl.dataflow.dflow DEBUG: Task 1155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1155 has memoization hash 19fcc990062dcf3007bb5a7b58e69d84 parsl.dataflow.memoization INFO: Task 1155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1155 try 0 launched on executor multi with executor id 1156 parsl.dataflow.dflow INFO: Standard output for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/2fe3007c-e543-4525-92b3-b4677e850574_isrForCrosstalkSources_1031725_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/2fe3007c-e543-4525-92b3-b4677e850574_isrForCrosstalkSources_1031725_13.stderr parsl.dataflow.dflow DEBUG: Task 1156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1156 has memoization hash 1d933fbfc52aa3be707122b6a633ffdc parsl.dataflow.memoization INFO: Task 1156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1156 try 0 launched on executor multi with executor id 1157 parsl.dataflow.dflow INFO: Standard output for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e28a358e-99f5-4d6b-ab00-acdb04200319_isrForCrosstalkSources_1031706_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e28a358e-99f5-4d6b-ab00-acdb04200319_isrForCrosstalkSources_1031706_30.stderr parsl.dataflow.dflow DEBUG: Task 1157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1157 has memoization hash 860030ff8628e5bda6c476d6b7aac879 parsl.dataflow.memoization INFO: Task 1157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1157 try 0 launched on executor multi with executor id 1158 parsl.dataflow.dflow INFO: Standard output for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/260bcf4c-6c4f-4e86-b5d5-b2d887d03a72_isrForCrosstalkSources_1031738_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/260bcf4c-6c4f-4e86-b5d5-b2d887d03a72_isrForCrosstalkSources_1031738_42.stderr parsl.dataflow.dflow DEBUG: Task 1158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1158 has memoization hash 099e16dc05b1bdd0b360318e0b26717b parsl.dataflow.memoization INFO: Task 1158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1158 try 0 launched on executor multi with executor id 1159 parsl.dataflow.dflow INFO: Standard output for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5cd30635-b101-409d-9610-fbd9f71149af_isrForCrosstalkSources_1031735_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5cd30635-b101-409d-9610-fbd9f71149af_isrForCrosstalkSources_1031735_38.stderr parsl.dataflow.dflow DEBUG: Task 1159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1159 has memoization hash d9eb06ead68703c80a56c17251dd7ede parsl.dataflow.memoization INFO: Task 1159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1159 try 0 launched on executor multi with executor id 1160 parsl.dataflow.dflow INFO: Standard output for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/be5be05d-0a48-420f-bfb6-72c021ea5e06_isrForCrosstalkSources_1031702_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/be5be05d-0a48-420f-bfb6-72c021ea5e06_isrForCrosstalkSources_1031702_2.stderr parsl.dataflow.dflow DEBUG: Task 1160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1160 has memoization hash 78d4b0101d1a3c9ae4bf9395552aa878 parsl.dataflow.memoization INFO: Task 1160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1160 try 0 launched on executor multi with executor id 1161 parsl.dataflow.dflow INFO: Standard output for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e565477c-a189-4c71-a7ad-cfe2c76c8aee_isrForCrosstalkSources_1031727_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e565477c-a189-4c71-a7ad-cfe2c76c8aee_isrForCrosstalkSources_1031727_40.stderr parsl.dataflow.dflow DEBUG: Task 1161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1161 has memoization hash 4193ae0a6579c8757e1157ff9e7f718e parsl.dataflow.memoization INFO: Task 1161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1161 try 0 launched on executor multi with executor id 1162 parsl.dataflow.dflow INFO: Standard output for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/cdad2412-a95e-4f21-bdb3-5f7182c65574_isrForCrosstalkSources_1031722_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/cdad2412-a95e-4f21-bdb3-5f7182c65574_isrForCrosstalkSources_1031722_26.stderr parsl.dataflow.dflow DEBUG: Task 1162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1162 has memoization hash 426fd5631ce2a665b49fd6ffcc5a1232 parsl.dataflow.memoization INFO: Task 1162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1162 try 0 launched on executor multi with executor id 1163 parsl.dataflow.dflow INFO: Standard output for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/c869d061-ea41-4089-b978-4fb4802a0a3c_isrForCrosstalkSources_1031737_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/c869d061-ea41-4089-b978-4fb4802a0a3c_isrForCrosstalkSources_1031737_36.stderr parsl.dataflow.dflow DEBUG: Task 1163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1163 has memoization hash f018eb9646df9fb608f3ee092b3ff0a5 parsl.dataflow.memoization INFO: Task 1163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1163 try 0 launched on executor multi with executor id 1164 parsl.dataflow.dflow INFO: Standard output for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/97e027d8-0c09-4e46-81a7-f8a9749a5c6a_isrForCrosstalkSources_1031700_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/97e027d8-0c09-4e46-81a7-f8a9749a5c6a_isrForCrosstalkSources_1031700_37.stderr parsl.dataflow.dflow DEBUG: Task 1164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1164 has memoization hash 9db82370995e05ad06e1dd361981d2e1 parsl.dataflow.memoization INFO: Task 1164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1164 try 0 launched on executor multi with executor id 1165 parsl.dataflow.dflow INFO: Standard output for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/5ba37f56-9c38-413f-bd7a-068d408444b3_isrForCrosstalkSources_1031725_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/5ba37f56-9c38-413f-bd7a-068d408444b3_isrForCrosstalkSources_1031725_6.stderr parsl.dataflow.dflow DEBUG: Task 1165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1165 has memoization hash 3b2fca4951313a1a110ab7b7d2f47e30 parsl.dataflow.memoization INFO: Task 1165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1165 try 0 launched on executor multi with executor id 1166 parsl.dataflow.dflow INFO: Standard output for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f1fdb75c-4e86-46e9-945a-1eaf02a2b741_isrForCrosstalkSources_1031709_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f1fdb75c-4e86-46e9-945a-1eaf02a2b741_isrForCrosstalkSources_1031709_33.stderr parsl.dataflow.dflow DEBUG: Task 1166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1166 has memoization hash ccb4341449cabcfcc592dbf65d77033a parsl.dataflow.memoization INFO: Task 1166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcabba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1166 try 0 launched on executor multi with executor id 1167 parsl.dataflow.dflow INFO: Standard output for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/9ec0b6a0-2ed4-4d81-beaf-de59b2ee4297_isrForCrosstalkSources_1031737_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/9ec0b6a0-2ed4-4d81-beaf-de59b2ee4297_isrForCrosstalkSources_1031737_27.stderr parsl.dataflow.dflow DEBUG: Task 1167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1167 has memoization hash e199027f94782809b59ecdbdef4a66eb parsl.dataflow.memoization INFO: Task 1167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcabce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1167 try 0 launched on executor multi with executor id 1168 parsl.dataflow.dflow INFO: Standard output for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9aae791b-33eb-4096-96d7-904fa8d909f2_isrForCrosstalkSources_1031718_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9aae791b-33eb-4096-96d7-904fa8d909f2_isrForCrosstalkSources_1031718_34.stderr parsl.dataflow.dflow DEBUG: Task 1168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1168 has memoization hash cb2659d7f04d885bb5da448e78706560 parsl.dataflow.memoization INFO: Task 1168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcabb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1168 try 0 launched on executor multi with executor id 1169 parsl.dataflow.dflow INFO: Standard output for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/133fd46b-0eeb-4b82-970d-c8809585f2b5_isrForCrosstalkSources_1031731_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/133fd46b-0eeb-4b82-970d-c8809585f2b5_isrForCrosstalkSources_1031731_50.stderr parsl.dataflow.dflow DEBUG: Task 1169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1169 has memoization hash 543dcc4b04bc365a292543215e69ebca parsl.dataflow.memoization INFO: Task 1169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcabc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1169 try 0 launched on executor multi with executor id 1170 parsl.dataflow.dflow INFO: Standard output for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/097df3b1-04fa-4f6a-afd1-9b5dc7ce137b_isrForCrosstalkSources_1031710_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/097df3b1-04fa-4f6a-afd1-9b5dc7ce137b_isrForCrosstalkSources_1031710_29.stderr parsl.dataflow.dflow DEBUG: Task 1170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1170 has memoization hash 2a9c818592b3b9e9c638f57d7fc5d008 parsl.dataflow.memoization INFO: Task 1170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcabe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1170 try 0 launched on executor multi with executor id 1171 parsl.dataflow.dflow INFO: Standard output for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3f8e18c5-9ef9-4c98-8c74-78dffff4ca02_isrForCrosstalkSources_1031738_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3f8e18c5-9ef9-4c98-8c74-78dffff4ca02_isrForCrosstalkSources_1031738_56.stderr parsl.dataflow.dflow DEBUG: Task 1171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1171 has memoization hash 791e418e376d38b289db9a94d8dc4ed2 parsl.dataflow.memoization INFO: Task 1171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcabec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1171 try 0 launched on executor multi with executor id 1172 parsl.dataflow.dflow INFO: Standard output for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d88df21e-112e-4518-8868-62cbf4eb05dd_isrForCrosstalkSources_1031738_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d88df21e-112e-4518-8868-62cbf4eb05dd_isrForCrosstalkSources_1031738_60.stderr parsl.dataflow.dflow DEBUG: Task 1172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1172 has memoization hash 78b58731db75dfe2ebd68df98d500efe parsl.dataflow.memoization INFO: Task 1172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcabf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1172 try 0 launched on executor multi with executor id 1173 parsl.dataflow.dflow INFO: Standard output for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8f4122dc-b74a-47bd-869a-1d5455ae90cb_isrForCrosstalkSources_1031703_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8f4122dc-b74a-47bd-869a-1d5455ae90cb_isrForCrosstalkSources_1031703_25.stderr parsl.dataflow.dflow DEBUG: Task 1173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1173 has memoization hash 3acafdce96c9d496dfcdb9f8aaa3612c parsl.dataflow.memoization INFO: Task 1173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcabd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1173 try 0 launched on executor multi with executor id 1174 parsl.dataflow.dflow INFO: Standard output for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1631fbb5-4ffe-4087-a212-427654300aa2_isrForCrosstalkSources_1031711_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1631fbb5-4ffe-4087-a212-427654300aa2_isrForCrosstalkSources_1031711_28.stderr parsl.dataflow.dflow DEBUG: Task 1174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1174 has memoization hash f15cdde5681194d8cbf62a5a1306475f parsl.dataflow.memoization INFO: Task 1174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1174 try 0 launched on executor multi with executor id 1175 parsl.dataflow.dflow INFO: Standard output for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/a90b419a-df79-4645-8ae6-889b878f9c98_isrForCrosstalkSources_1031705_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/a90b419a-df79-4645-8ae6-889b878f9c98_isrForCrosstalkSources_1031705_42.stderr parsl.dataflow.dflow DEBUG: Task 1175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1175 has memoization hash 4b61bd93c8b9686a36058b8494eb48b2 parsl.dataflow.memoization INFO: Task 1175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb040e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1175 try 0 launched on executor multi with executor id 1176 parsl.dataflow.dflow INFO: Standard output for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a7869cf1-8ba4-471f-a0f8-ad250473f5c9_isrForCrosstalkSources_1031706_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a7869cf1-8ba4-471f-a0f8-ad250473f5c9_isrForCrosstalkSources_1031706_7.stderr parsl.dataflow.dflow DEBUG: Task 1176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1176 has memoization hash a265f856d8a9d060fca130eefa9288cf parsl.dataflow.memoization INFO: Task 1176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1176 try 0 launched on executor multi with executor id 1177 parsl.dataflow.dflow INFO: Standard output for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/9b3fff15-a0d0-4c4f-84c8-fd4083ebec50_isrForCrosstalkSources_1031738_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/9b3fff15-a0d0-4c4f-84c8-fd4083ebec50_isrForCrosstalkSources_1031738_26.stderr parsl.dataflow.dflow DEBUG: Task 1177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1177 has memoization hash 3560587fd79e7b8368d74c84027d2b6e parsl.dataflow.memoization INFO: Task 1177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb042c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1177 try 0 launched on executor multi with executor id 1178 parsl.dataflow.dflow INFO: Standard output for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/797b01fd-033d-4a9b-a966-50a04090d9f8_isrForCrosstalkSources_1031736_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/797b01fd-033d-4a9b-a966-50a04090d9f8_isrForCrosstalkSources_1031736_41.stderr parsl.dataflow.dflow DEBUG: Task 1178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1178 has memoization hash cd12ca6aa64d99056b8528930996dfa3 parsl.dataflow.memoization INFO: Task 1178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1178 try 0 launched on executor multi with executor id 1179 parsl.dataflow.dflow INFO: Standard output for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/cc7ef2ef-47e3-46e2-915c-bbe0e941b7ad_isrForCrosstalkSources_1031718_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/cc7ef2ef-47e3-46e2-915c-bbe0e941b7ad_isrForCrosstalkSources_1031718_62.stderr parsl.dataflow.dflow DEBUG: Task 1179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1179 has memoization hash 15775b2de295c42338c7a07a82c99904 parsl.dataflow.memoization INFO: Task 1179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1179 try 0 launched on executor multi with executor id 1180 parsl.dataflow.dflow INFO: Standard output for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c853fdf4-9e13-43c2-a77e-65fca2dc50b0_isrForCrosstalkSources_1031732_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c853fdf4-9e13-43c2-a77e-65fca2dc50b0_isrForCrosstalkSources_1031732_59.stderr parsl.dataflow.dflow DEBUG: Task 1180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1180 has memoization hash ca41c86d291b31ee6c4e3fd948e7c194 parsl.dataflow.memoization INFO: Task 1180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1180 try 0 launched on executor multi with executor id 1181 parsl.dataflow.dflow INFO: Standard output for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4c5d2dd1-3eb2-468c-891c-6f4de05d7a82_isrForCrosstalkSources_1031719_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4c5d2dd1-3eb2-468c-891c-6f4de05d7a82_isrForCrosstalkSources_1031719_52.stderr parsl.dataflow.dflow DEBUG: Task 1181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1181 has memoization hash 7c24d876e277d52dbebbf877f346b5a5 parsl.dataflow.memoization INFO: Task 1181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb044a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1181 try 0 launched on executor multi with executor id 1182 parsl.dataflow.dflow INFO: Standard output for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/54516b10-bb09-43dc-8da4-d68052cc1009_isrForCrosstalkSources_1031728_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/54516b10-bb09-43dc-8da4-d68052cc1009_isrForCrosstalkSources_1031728_46.stderr parsl.dataflow.dflow DEBUG: Task 1182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1182 has memoization hash 75240c42b9b63f43634098fdab08d8ac parsl.dataflow.memoization INFO: Task 1182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb045e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1182 try 0 launched on executor multi with executor id 1183 parsl.dataflow.dflow INFO: Standard output for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/22a6ee97-64c2-43dd-911b-d9058602700e_isrForCrosstalkSources_1031700_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/22a6ee97-64c2-43dd-911b-d9058602700e_isrForCrosstalkSources_1031700_57.stderr parsl.dataflow.dflow DEBUG: Task 1183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1183 has memoization hash ee3c07cc2d160edfc86f944829a1cda2 parsl.dataflow.memoization INFO: Task 1183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1183 try 0 launched on executor multi with executor id 1184 parsl.dataflow.dflow INFO: Standard output for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d9e621e4-1585-4d12-bdf5-ad51a6b16639_isrForCrosstalkSources_1031703_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d9e621e4-1585-4d12-bdf5-ad51a6b16639_isrForCrosstalkSources_1031703_19.stderr parsl.dataflow.dflow DEBUG: Task 1184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1184 has memoization hash 7f2bcfa1b2e45754b413f6cf7268a55a parsl.dataflow.memoization INFO: Task 1184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1184 try 0 launched on executor multi with executor id 1185 parsl.dataflow.dflow INFO: Standard output for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/99aabce6-4c08-4a98-9f78-1c12ca2bda2d_isrForCrosstalkSources_1031716_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/99aabce6-4c08-4a98-9f78-1c12ca2bda2d_isrForCrosstalkSources_1031716_47.stderr parsl.dataflow.dflow DEBUG: Task 1185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1185 has memoization hash a737e3e523094768636c18e6573ff5c0 parsl.dataflow.memoization INFO: Task 1185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1185 try 0 launched on executor multi with executor id 1186 parsl.dataflow.dflow INFO: Standard output for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ebaa52d9-6ddb-4d7b-8846-516835928303_isrForCrosstalkSources_1031708_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ebaa52d9-6ddb-4d7b-8846-516835928303_isrForCrosstalkSources_1031708_39.stderr parsl.dataflow.dflow DEBUG: Task 1186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1186 has memoization hash bab51d9ef72f0c9410f8d42088d327b5 parsl.dataflow.memoization INFO: Task 1186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb047c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1186 try 0 launched on executor multi with executor id 1187 parsl.dataflow.dflow INFO: Standard output for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/836afdc6-28c3-4d5b-b192-909535e0e424_isrForCrosstalkSources_1031711_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/836afdc6-28c3-4d5b-b192-909535e0e424_isrForCrosstalkSources_1031711_53.stderr parsl.dataflow.dflow DEBUG: Task 1187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1187 has memoization hash 8b8144e0440558ef3cef4e8cac0cedc3 parsl.dataflow.memoization INFO: Task 1187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1187 try 0 launched on executor multi with executor id 1188 parsl.dataflow.dflow INFO: Standard output for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/20dd828f-0f2b-468b-8175-31fdcff93ac5_isrForCrosstalkSources_1031712_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/20dd828f-0f2b-468b-8175-31fdcff93ac5_isrForCrosstalkSources_1031712_57.stderr parsl.dataflow.dflow DEBUG: Task 1188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1188 has memoization hash 93b9ed1bcb242fc61f1ffe3b3b59f3ac parsl.dataflow.memoization INFO: Task 1188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1188 try 0 launched on executor multi with executor id 1189 parsl.dataflow.dflow INFO: Standard output for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f92f5092-9e06-4560-a4ae-93c1212ec3e8_isrForCrosstalkSources_1031705_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f92f5092-9e06-4560-a4ae-93c1212ec3e8_isrForCrosstalkSources_1031705_38.stderr parsl.dataflow.dflow DEBUG: Task 1189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1189 has memoization hash e95fc30519d2cc50dc1d47be01252f87 parsl.dataflow.memoization INFO: Task 1189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1189 try 0 launched on executor multi with executor id 1190 parsl.dataflow.dflow INFO: Standard output for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/74b5fc25-5f4e-4557-b35b-25c97608c454_isrForCrosstalkSources_1031723_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/74b5fc25-5f4e-4557-b35b-25c97608c454_isrForCrosstalkSources_1031723_3.stderr parsl.dataflow.dflow DEBUG: Task 1190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1190 has memoization hash dcb426937930aa2f906ce747465c4041 parsl.dataflow.memoization INFO: Task 1190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1190 try 0 launched on executor multi with executor id 1191 parsl.dataflow.dflow INFO: Standard output for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/ca67070d-6264-4f9b-85cf-54d180e0b221_isrForCrosstalkSources_1031705_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/ca67070d-6264-4f9b-85cf-54d180e0b221_isrForCrosstalkSources_1031705_62.stderr parsl.dataflow.dflow DEBUG: Task 1191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1191 has memoization hash 4750333c7ded0ca3be59f96d0106e04d parsl.dataflow.memoization INFO: Task 1191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1191 try 0 launched on executor multi with executor id 1192 parsl.dataflow.dflow INFO: Standard output for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d99c8b25-2c4d-4c56-96ec-a430129dec4b_isrForCrosstalkSources_1031722_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d99c8b25-2c4d-4c56-96ec-a430129dec4b_isrForCrosstalkSources_1031722_38.stderr parsl.dataflow.dflow DEBUG: Task 1192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1192 has memoization hash a0121b6be891164e66a8f3fc1d908e00 parsl.dataflow.memoization INFO: Task 1192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1192 try 0 launched on executor multi with executor id 1193 parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ec2a45be-e1ae-47a7-998a-fd5aa71a91c1_isrForCrosstalkSources_1031708_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ec2a45be-e1ae-47a7-998a-fd5aa71a91c1_isrForCrosstalkSources_1031708_59.stderr parsl.dataflow.dflow DEBUG: Task 1193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1193 has memoization hash b0d7c9694b3a21bc757473e84b91b154 parsl.dataflow.memoization INFO: Task 1193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1193 try 0 launched on executor multi with executor id 1194 parsl.dataflow.dflow INFO: Standard output for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6905f6db-1113-4a2d-8422-062ef5ed7db3_isrForCrosstalkSources_1031700_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6905f6db-1113-4a2d-8422-062ef5ed7db3_isrForCrosstalkSources_1031700_5.stderr parsl.dataflow.dflow DEBUG: Task 1194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1194 has memoization hash df86ad577af0b62425ce74dda3756580 parsl.dataflow.memoization INFO: Task 1194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1194 try 0 launched on executor multi with executor id 1195 parsl.dataflow.dflow INFO: Standard output for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/59020227-c6db-4ac0-8b56-1f9d2aa8191f_isrForCrosstalkSources_1031727_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/59020227-c6db-4ac0-8b56-1f9d2aa8191f_isrForCrosstalkSources_1031727_49.stderr parsl.dataflow.dflow DEBUG: Task 1195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1195 has memoization hash 5796a629984ecaed95d8e47ba2ac647e parsl.dataflow.memoization INFO: Task 1195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1195 try 0 launched on executor multi with executor id 1196 parsl.dataflow.dflow INFO: Standard output for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/97e02201-4ad0-487d-a2dc-b83767f40b25_isrForCrosstalkSources_1031711_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/97e02201-4ad0-487d-a2dc-b83767f40b25_isrForCrosstalkSources_1031711_8.stderr parsl.dataflow.dflow DEBUG: Task 1196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1196 has memoization hash 24fd469456e24e2027b26b0c6fdec66e parsl.dataflow.memoization INFO: Task 1196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1196 try 0 launched on executor multi with executor id 1197 parsl.dataflow.dflow INFO: Standard output for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/903e1325-4ebf-43a0-aa66-14be3034d212_isrForCrosstalkSources_1031710_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/903e1325-4ebf-43a0-aa66-14be3034d212_isrForCrosstalkSources_1031710_50.stderr parsl.dataflow.dflow DEBUG: Task 1197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1197 has memoization hash f4ce803f1994227b522e463971e45e56 parsl.dataflow.memoization INFO: Task 1197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1197 try 0 launched on executor multi with executor id 1198 parsl.dataflow.dflow INFO: Standard output for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/7847e7cb-4b0a-47ba-9083-41e7b52e514b_isrForCrosstalkSources_1031726_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/7847e7cb-4b0a-47ba-9083-41e7b52e514b_isrForCrosstalkSources_1031726_45.stderr parsl.dataflow.dflow DEBUG: Task 1198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1198 has memoization hash 8edc29a3392ee7701ba7de18e1159136 parsl.dataflow.memoization INFO: Task 1198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1198 try 0 launched on executor multi with executor id 1199 parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0d8f1e8c-3c97-437e-9205-3dd24b3db345_isrForCrosstalkSources_1031711_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0d8f1e8c-3c97-437e-9205-3dd24b3db345_isrForCrosstalkSources_1031711_3.stderr parsl.dataflow.dflow DEBUG: Task 1199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1199 has memoization hash 0b9e0a9fc9a05ac25def6a73f442c8c9 parsl.dataflow.memoization INFO: Task 1199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1199 try 0 launched on executor multi with executor id 1200 parsl.dataflow.dflow INFO: Standard output for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/67d111ff-f398-4d73-a8bd-4768611b6386_isrForCrosstalkSources_1031725_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/67d111ff-f398-4d73-a8bd-4768611b6386_isrForCrosstalkSources_1031725_42.stderr parsl.dataflow.dflow DEBUG: Task 1200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1200 has memoization hash 3861cdc7e5f27049aba21480876eae39 parsl.dataflow.memoization INFO: Task 1200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1200 try 0 launched on executor multi with executor id 1201 parsl.dataflow.dflow INFO: Standard output for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/631399f1-ae03-423c-9ae6-db9a2b0cbd43_isrForCrosstalkSources_1031702_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/631399f1-ae03-423c-9ae6-db9a2b0cbd43_isrForCrosstalkSources_1031702_28.stderr parsl.dataflow.dflow DEBUG: Task 1201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1201 has memoization hash fb38d68fb6a464168831cf39333656e5 parsl.dataflow.memoization INFO: Task 1201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1201 try 0 launched on executor multi with executor id 1202 parsl.dataflow.dflow INFO: Standard output for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/b5492ec0-4493-4c7f-afe7-2de64b442b18_isrForCrosstalkSources_1031734_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/b5492ec0-4493-4c7f-afe7-2de64b442b18_isrForCrosstalkSources_1031734_53.stderr parsl.dataflow.dflow DEBUG: Task 1202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1202 has memoization hash 7b8190d99dd559c0eb72c51cf3940fee parsl.dataflow.memoization INFO: Task 1202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1202 try 0 launched on executor multi with executor id 1203 parsl.dataflow.dflow INFO: Standard output for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d20dda54-641c-4038-9b46-da37295dcd4b_isrForCrosstalkSources_1031711_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d20dda54-641c-4038-9b46-da37295dcd4b_isrForCrosstalkSources_1031711_59.stderr parsl.dataflow.dflow DEBUG: Task 1203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1203 has memoization hash 24a97159c6ab835704428ed325bcc3b6 parsl.dataflow.memoization INFO: Task 1203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb051c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1203 try 0 launched on executor multi with executor id 1204 parsl.dataflow.dflow INFO: Standard output for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/26d0d23c-eaf1-4198-9530-9df50c758276_isrForCrosstalkSources_1031705_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/26d0d23c-eaf1-4198-9530-9df50c758276_isrForCrosstalkSources_1031705_43.stderr parsl.dataflow.dflow DEBUG: Task 1204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1204 has memoization hash c86cef50c393070ddf509aa3455a8035 parsl.dataflow.memoization INFO: Task 1204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1204 try 0 launched on executor multi with executor id 1205 parsl.dataflow.dflow INFO: Standard output for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6269d343-257a-4906-8c38-10dcf31f26fa_isrForCrosstalkSources_1031700_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6269d343-257a-4906-8c38-10dcf31f26fa_isrForCrosstalkSources_1031700_26.stderr parsl.dataflow.dflow DEBUG: Task 1205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1205 has memoization hash b29990cce2afc600cb0f6da83596e760 parsl.dataflow.memoization INFO: Task 1205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb053a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1205 try 0 launched on executor multi with executor id 1206 parsl.dataflow.dflow INFO: Standard output for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/75974c60-c5aa-457e-88fd-f9b6ba696336_isrForCrosstalkSources_1031723_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/75974c60-c5aa-457e-88fd-f9b6ba696336_isrForCrosstalkSources_1031723_55.stderr parsl.dataflow.dflow DEBUG: Task 1206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1206 has memoization hash 087abffa40fb455f8b698c951a557934 parsl.dataflow.memoization INFO: Task 1206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1206 try 0 launched on executor multi with executor id 1207 parsl.dataflow.dflow INFO: Standard output for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8fa2f3c4-8429-49f7-922c-325c92633f7f_isrForCrosstalkSources_1031703_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8fa2f3c4-8429-49f7-922c-325c92633f7f_isrForCrosstalkSources_1031703_48.stderr parsl.dataflow.dflow DEBUG: Task 1207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1207 has memoization hash 7fe1dd660552ee1917f6da97c9283bab parsl.dataflow.memoization INFO: Task 1207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1207 try 0 launched on executor multi with executor id 1208 parsl.dataflow.dflow INFO: Standard output for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/3f4cb789-91db-4113-beb0-885686fac6b2_isrForCrosstalkSources_1031717_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/3f4cb789-91db-4113-beb0-885686fac6b2_isrForCrosstalkSources_1031717_25.stderr parsl.dataflow.dflow DEBUG: Task 1208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1208 has memoization hash 9b333be2b66d10214cc538db78bdd651 parsl.dataflow.memoization INFO: Task 1208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb054e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1208 try 0 launched on executor multi with executor id 1209 parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8445e70a-de9c-4cc7-af0b-b357b4f9c904_isrForCrosstalkSources_1031704_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8445e70a-de9c-4cc7-af0b-b357b4f9c904_isrForCrosstalkSources_1031704_41.stderr parsl.dataflow.dflow DEBUG: Task 1209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1209 has memoization hash 0d3c112e06ecac6665efd334b0bed349 parsl.dataflow.memoization INFO: Task 1209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1209 try 0 launched on executor multi with executor id 1210 parsl.dataflow.dflow INFO: Standard output for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4505e8bd-d9ec-4099-a448-99d14f8e2a89_isrForCrosstalkSources_1031706_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4505e8bd-d9ec-4099-a448-99d14f8e2a89_isrForCrosstalkSources_1031706_4.stderr parsl.dataflow.dflow DEBUG: Task 1210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1210 has memoization hash 7dd8b2864d61e580904de97f7d6bf885 parsl.dataflow.memoization INFO: Task 1210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1210 try 0 launched on executor multi with executor id 1211 parsl.dataflow.dflow INFO: Standard output for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a251471b-7369-4b06-83ca-88134f081710_isrForCrosstalkSources_1031718_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a251471b-7369-4b06-83ca-88134f081710_isrForCrosstalkSources_1031718_43.stderr parsl.dataflow.dflow DEBUG: Task 1211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1211 has memoization hash a3421603c4d057fba113d4dfdd04a316 parsl.dataflow.memoization INFO: Task 1211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1211 try 0 launched on executor multi with executor id 1212 parsl.dataflow.dflow INFO: Standard output for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4cdce307-0563-4156-9d54-4407caf5e4a9_isrForCrosstalkSources_1031717_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4cdce307-0563-4156-9d54-4407caf5e4a9_isrForCrosstalkSources_1031717_24.stderr parsl.dataflow.dflow DEBUG: Task 1212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1212 has memoization hash 38753bb3175ba30959e6cd1e68134cf7 parsl.dataflow.memoization INFO: Task 1212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb058a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1212 try 0 launched on executor multi with executor id 1213 parsl.dataflow.dflow INFO: Standard output for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/a3fb240a-714c-4477-af1d-d2d348664b92_isrForCrosstalkSources_1031699_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/a3fb240a-714c-4477-af1d-d2d348664b92_isrForCrosstalkSources_1031699_7.stderr parsl.dataflow.dflow DEBUG: Task 1213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1213 has memoization hash 0e3eb903f4b16fe953decd2d3ac518e1 parsl.dataflow.memoization INFO: Task 1213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1213 try 0 launched on executor multi with executor id 1214 parsl.dataflow.dflow INFO: Standard output for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4bbf1243-d1d9-4fc5-a53d-130f751e113b_isrForCrosstalkSources_1031718_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4bbf1243-d1d9-4fc5-a53d-130f751e113b_isrForCrosstalkSources_1031718_25.stderr parsl.dataflow.dflow DEBUG: Task 1214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1214 has memoization hash 76b2a2b764f016bec8a6b8887367ffa0 parsl.dataflow.memoization INFO: Task 1214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1214 try 0 launched on executor multi with executor id 1215 parsl.dataflow.dflow INFO: Standard output for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/19357293-f5af-4b98-a551-68f1f4fcd2b3_isrForCrosstalkSources_1031710_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/19357293-f5af-4b98-a551-68f1f4fcd2b3_isrForCrosstalkSources_1031710_21.stderr parsl.dataflow.dflow DEBUG: Task 1215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1215 has memoization hash 396cfc2150c7f50ccd823fbc77ecf828 parsl.dataflow.memoization INFO: Task 1215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb059e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1215 try 0 launched on executor multi with executor id 1216 parsl.dataflow.dflow INFO: Standard output for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/7e9d0ec8-087a-4528-bd58-28159e07461b_isrForCrosstalkSources_1031709_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/7e9d0ec8-087a-4528-bd58-28159e07461b_isrForCrosstalkSources_1031709_10.stderr parsl.dataflow.dflow DEBUG: Task 1216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1216 has memoization hash fd583e3c855759b7c52cad6ebac881e1 parsl.dataflow.memoization INFO: Task 1216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1216 try 0 launched on executor multi with executor id 1217 parsl.dataflow.dflow INFO: Standard output for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5ff62239-594c-4c03-a3d6-446ef4e82f50_isrForCrosstalkSources_1031705_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5ff62239-594c-4c03-a3d6-446ef4e82f50_isrForCrosstalkSources_1031705_41.stderr parsl.dataflow.dflow DEBUG: Task 1217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1217 has memoization hash c3172f4a07d6498f7c5f803d111f2db6 parsl.dataflow.memoization INFO: Task 1217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1217 try 0 launched on executor multi with executor id 1218 parsl.dataflow.dflow INFO: Standard output for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/615dfcfd-f669-4e7a-a048-ec899d01fb99_isrForCrosstalkSources_1031717_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/615dfcfd-f669-4e7a-a048-ec899d01fb99_isrForCrosstalkSources_1031717_41.stderr parsl.dataflow.dflow DEBUG: Task 1218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1218 has memoization hash 1278c4a73b58efcae2a898386455adf9 parsl.dataflow.memoization INFO: Task 1218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1218 try 0 launched on executor multi with executor id 1219 parsl.dataflow.dflow INFO: Standard output for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f9757a61-a497-4bfd-915d-3b9ba9932f64_isrForCrosstalkSources_1031710_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f9757a61-a497-4bfd-915d-3b9ba9932f64_isrForCrosstalkSources_1031710_62.stderr parsl.dataflow.dflow DEBUG: Task 1219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1219 has memoization hash 216b98d281792e85122181e2fb1e7707 parsl.dataflow.memoization INFO: Task 1219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1219 try 0 launched on executor multi with executor id 1220 parsl.dataflow.dflow INFO: Standard output for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c656dc57-7c40-4c06-8d97-0c3ffbd9ef30_isrForCrosstalkSources_1031706_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c656dc57-7c40-4c06-8d97-0c3ffbd9ef30_isrForCrosstalkSources_1031706_22.stderr parsl.dataflow.dflow DEBUG: Task 1220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1220 has memoization hash ba5283c13d27626ec28cd223eafc79a2 parsl.dataflow.memoization INFO: Task 1220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1220 try 0 launched on executor multi with executor id 1221 parsl.dataflow.dflow INFO: Standard output for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/464a95e8-3b5b-48a7-b42c-afaaaedc8a4c_isrForCrosstalkSources_1031735_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/464a95e8-3b5b-48a7-b42c-afaaaedc8a4c_isrForCrosstalkSources_1031735_20.stderr parsl.dataflow.dflow DEBUG: Task 1221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1221 has memoization hash 1f88690e14f446b076fdeeaa6a3be9f9 parsl.dataflow.memoization INFO: Task 1221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1221 try 0 launched on executor multi with executor id 1222 parsl.dataflow.dflow INFO: Standard output for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e74c570b-0254-4b5d-8ee2-7c60c2615477_isrForCrosstalkSources_1031704_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e74c570b-0254-4b5d-8ee2-7c60c2615477_isrForCrosstalkSources_1031704_33.stderr parsl.dataflow.dflow DEBUG: Task 1222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1222 has memoization hash a5eb5b4d0db5300eac009e24399b22d5 parsl.dataflow.memoization INFO: Task 1222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1222 try 0 launched on executor multi with executor id 1223 parsl.dataflow.dflow INFO: Standard output for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/9d1e2683-0207-4913-8387-c255788306da_isrForCrosstalkSources_1031708_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/9d1e2683-0207-4913-8387-c255788306da_isrForCrosstalkSources_1031708_5.stderr parsl.dataflow.dflow DEBUG: Task 1223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1223 has memoization hash f107fc27c820cd0721d169d023bea10b parsl.dataflow.memoization INFO: Task 1223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1223 try 0 launched on executor multi with executor id 1224 parsl.dataflow.dflow INFO: Standard output for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/30785a45-24d9-4fad-b8c2-8acf09b11224_isrForCrosstalkSources_1031702_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/30785a45-24d9-4fad-b8c2-8acf09b11224_isrForCrosstalkSources_1031702_25.stderr parsl.dataflow.dflow DEBUG: Task 1224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1224 has memoization hash 647ef3cb581e9c287a8135247918733e parsl.dataflow.memoization INFO: Task 1224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb05f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1224 try 0 launched on executor multi with executor id 1225 parsl.dataflow.dflow INFO: Standard output for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/7f203e98-715a-4ed6-b61b-1082d7ab18ce_isrForCrosstalkSources_1031728_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/7f203e98-715a-4ed6-b61b-1082d7ab18ce_isrForCrosstalkSources_1031728_11.stderr parsl.dataflow.dflow DEBUG: Task 1225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1225 has memoization hash 7e6b9ae9543a0fdaa887ca473a3bf9f2 parsl.dataflow.memoization INFO: Task 1225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1225 try 0 launched on executor multi with executor id 1226 parsl.dataflow.dflow INFO: Standard output for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/72e8b132-0031-4b07-9e5d-9c4cdb84774b_isrForCrosstalkSources_1031716_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/72e8b132-0031-4b07-9e5d-9c4cdb84774b_isrForCrosstalkSources_1031716_43.stderr parsl.dataflow.dflow DEBUG: Task 1226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1226 has memoization hash 285e6722c33ff1a8985a774a8a9edcb8 parsl.dataflow.memoization INFO: Task 1226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb060c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1226 try 0 launched on executor multi with executor id 1227 parsl.dataflow.dflow INFO: Standard output for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/fddbc86d-8069-49b8-ba9c-8e7b2ac5e1a0_isrForCrosstalkSources_1031719_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/fddbc86d-8069-49b8-ba9c-8e7b2ac5e1a0_isrForCrosstalkSources_1031719_33.stderr parsl.dataflow.dflow DEBUG: Task 1227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1227 has memoization hash 87c8753c0f5525d5ee2aa4087de95cc1 parsl.dataflow.memoization INFO: Task 1227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1227 try 0 launched on executor multi with executor id 1228 parsl.dataflow.dflow INFO: Standard output for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8cb8e2e4-0377-4902-89f1-de2a06ca876e_isrForCrosstalkSources_1031700_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8cb8e2e4-0377-4902-89f1-de2a06ca876e_isrForCrosstalkSources_1031700_51.stderr parsl.dataflow.dflow DEBUG: Task 1228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1228 has memoization hash ee4d674b4fd0edaa897d1787df9f157a parsl.dataflow.memoization INFO: Task 1228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1228 try 0 launched on executor multi with executor id 1229 parsl.dataflow.dflow INFO: Standard output for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/fa3de3fd-f8a7-4061-82ce-8b9e45cbac5d_isrForCrosstalkSources_1031714_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/fa3de3fd-f8a7-4061-82ce-8b9e45cbac5d_isrForCrosstalkSources_1031714_43.stderr parsl.dataflow.dflow DEBUG: Task 1229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1229 has memoization hash 4a68adf1c6bf5c6ae953621ffed184d0 parsl.dataflow.memoization INFO: Task 1229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1229 try 0 launched on executor multi with executor id 1230 parsl.dataflow.dflow INFO: Standard output for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/1b637027-1fab-41c8-a203-a0aa9c998160_isrForCrosstalkSources_1031710_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/1b637027-1fab-41c8-a203-a0aa9c998160_isrForCrosstalkSources_1031710_38.stderr parsl.dataflow.dflow DEBUG: Task 1230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1230 has memoization hash 72ff52aaecb69326cf9e6ba9278d932e parsl.dataflow.memoization INFO: Task 1230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1230 try 0 launched on executor multi with executor id 1231 parsl.dataflow.dflow INFO: Standard output for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/89e0e916-480c-4641-a4dc-0ad61b53fd7d_isrForCrosstalkSources_1031713_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/89e0e916-480c-4641-a4dc-0ad61b53fd7d_isrForCrosstalkSources_1031713_41.stderr parsl.dataflow.dflow DEBUG: Task 1231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1231 has memoization hash fe88daea0e4424f2fa711b297f827e30 parsl.dataflow.memoization INFO: Task 1231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1231 try 0 launched on executor multi with executor id 1232 parsl.dataflow.dflow INFO: Standard output for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7202dedf-be56-4dd4-8efe-be872cbc00f1_isrForCrosstalkSources_1031702_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7202dedf-be56-4dd4-8efe-be872cbc00f1_isrForCrosstalkSources_1031702_13.stderr parsl.dataflow.dflow DEBUG: Task 1232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1232 has memoization hash bd5a1e0d202fc67b33d1316db97cf464 parsl.dataflow.memoization INFO: Task 1232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1232 try 0 launched on executor multi with executor id 1233 parsl.dataflow.dflow INFO: Standard output for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/c99273df-0dbc-454a-a640-4929c585e52f_isrForCrosstalkSources_1031709_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/c99273df-0dbc-454a-a640-4929c585e52f_isrForCrosstalkSources_1031709_17.stderr parsl.dataflow.dflow DEBUG: Task 1233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1233 has memoization hash a70d36fe1ec6705041aad8dab7220f41 parsl.dataflow.memoization INFO: Task 1233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1233 try 0 launched on executor multi with executor id 1234 parsl.dataflow.dflow INFO: Standard output for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/df9c587d-2e84-465c-bce2-8fdac255bc9a_isrForCrosstalkSources_1031699_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/df9c587d-2e84-465c-bce2-8fdac255bc9a_isrForCrosstalkSources_1031699_13.stderr parsl.dataflow.dflow DEBUG: Task 1234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1234 has memoization hash 09ef9c89a8c22c121a7e8cfc84eabfc2 parsl.dataflow.memoization INFO: Task 1234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb065c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1234 try 0 launched on executor multi with executor id 1235 parsl.dataflow.dflow INFO: Standard output for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/9cc89d63-6fa5-4f6a-88c7-5cddf3f7743e_isrForCrosstalkSources_1031723_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/9cc89d63-6fa5-4f6a-88c7-5cddf3f7743e_isrForCrosstalkSources_1031723_1.stderr parsl.dataflow.dflow DEBUG: Task 1235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1235 has memoization hash dfdde89503da8a3079b7a82eb81cfde4 parsl.dataflow.memoization INFO: Task 1235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1235 try 0 launched on executor multi with executor id 1236 parsl.dataflow.dflow INFO: Standard output for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/b0602fc4-e368-480a-a5c7-b9a857bc4f39_isrForCrosstalkSources_1031711_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/b0602fc4-e368-480a-a5c7-b9a857bc4f39_isrForCrosstalkSources_1031711_62.stderr parsl.dataflow.dflow DEBUG: Task 1236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1236 has memoization hash 89ec2ea8f18357481d7a741c87c04bb6 parsl.dataflow.memoization INFO: Task 1236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1236 try 0 launched on executor multi with executor id 1237 parsl.dataflow.dflow INFO: Standard output for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ed084f16-2a85-4aeb-8a1c-7f5709a67822_isrForCrosstalkSources_1031713_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ed084f16-2a85-4aeb-8a1c-7f5709a67822_isrForCrosstalkSources_1031713_52.stderr parsl.dataflow.dflow DEBUG: Task 1237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1237 has memoization hash b975036613bf30adf7c6c6e96f440fe8 parsl.dataflow.memoization INFO: Task 1237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1237 try 0 launched on executor multi with executor id 1238 parsl.dataflow.dflow INFO: Standard output for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/a22528dd-b849-425c-be0e-b46c71abbcb1_isrForCrosstalkSources_1031704_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/a22528dd-b849-425c-be0e-b46c71abbcb1_isrForCrosstalkSources_1031704_14.stderr parsl.dataflow.dflow DEBUG: Task 1238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1238 has memoization hash 4f9b152c90ac7ba9e2d8b024c3347d97 parsl.dataflow.memoization INFO: Task 1238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb067a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1238 try 0 launched on executor multi with executor id 1239 parsl.dataflow.dflow INFO: Standard output for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/bd80fefe-efb8-45ef-98db-840525a7c014_isrForCrosstalkSources_1031731_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/bd80fefe-efb8-45ef-98db-840525a7c014_isrForCrosstalkSources_1031731_15.stderr parsl.dataflow.dflow DEBUG: Task 1239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1239 has memoization hash 32104a1d9be7974523391fe0620830a4 parsl.dataflow.memoization INFO: Task 1239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:11 proc_lsst.multi:170 [INFO] [multi] job_id 17147947 proc_lsst.multi INFO: [multi] job_id 17147947 2024-03-17 04:02:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-17 04:02:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147947 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147947 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17147947 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-17 04:02:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-17 04:02:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-17 04:02:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1239 try 0 launched on executor multi with executor id 1240 parsl.dataflow.dflow INFO: Standard output for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/91ad8e44-58c5-47cc-b2fb-f86aa3343136_isrForCrosstalkSources_1031734_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/91ad8e44-58c5-47cc-b2fb-f86aa3343136_isrForCrosstalkSources_1031734_55.stderr parsl.dataflow.dflow DEBUG: Task 1240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1240 has memoization hash 9c35729a3269974a98ece56b58af4182 parsl.dataflow.memoization INFO: Task 1240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc45260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1240 try 0 launched on executor multi with executor id 1241 parsl.dataflow.dflow INFO: Standard output for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/e0302454-d593-4893-9879-0c8086ffeaf7_isrForCrosstalkSources_1031711_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/e0302454-d593-4893-9879-0c8086ffeaf7_isrForCrosstalkSources_1031711_16.stderr parsl.dataflow.dflow DEBUG: Task 1241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1241 has memoization hash 18017adebec6097051b3ee892c4075e1 parsl.dataflow.memoization INFO: Task 1241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fcaa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1241 try 0 launched on executor multi with executor id 1242 parsl.dataflow.dflow INFO: Standard output for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/a1522970-bbd9-4f42-a8ac-3f06c69fd58f_isrForCrosstalkSources_1031716_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/a1522970-bbd9-4f42-a8ac-3f06c69fd58f_isrForCrosstalkSources_1031716_27.stderr parsl.dataflow.dflow DEBUG: Task 1242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1242 has memoization hash 210a29956f5b5c9988e86388d25a4d3b parsl.dataflow.memoization INFO: Task 1242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc453a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1242 try 0 launched on executor multi with executor id 1243 parsl.dataflow.dflow INFO: Standard output for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/1d057f3c-aebe-44e1-865b-410bc77028ba_isrForCrosstalkSources_1031699_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/1d057f3c-aebe-44e1-865b-410bc77028ba_isrForCrosstalkSources_1031699_28.stderr parsl.dataflow.dflow DEBUG: Task 1243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1243 has memoization hash 477a8f8422fa8e7f3ad7d3bb19bd5ab6 parsl.dataflow.memoization INFO: Task 1243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb04040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1243 try 0 launched on executor multi with executor id 1244 parsl.dataflow.dflow INFO: Standard output for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bca6dc3e-270c-4174-b07b-f1f1db5643fb_isrForCrosstalkSources_1031723_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bca6dc3e-270c-4174-b07b-f1f1db5643fb_isrForCrosstalkSources_1031723_54.stderr parsl.dataflow.dflow DEBUG: Task 1244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1244 has memoization hash 4c142d0ab6108e571806c6e8060a7975 parsl.dataflow.memoization INFO: Task 1244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1244 try 0 launched on executor multi with executor id 1245 parsl.dataflow.dflow INFO: Standard output for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/aecc549c-53cf-4b3e-a7b2-9106702db5b0_isrForCrosstalkSources_1031721_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/aecc549c-53cf-4b3e-a7b2-9106702db5b0_isrForCrosstalkSources_1031721_7.stderr parsl.dataflow.dflow DEBUG: Task 1245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1245 has memoization hash f31bf67ca1d7289cb41822c3e7b891e9 parsl.dataflow.memoization INFO: Task 1245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1245 try 0 launched on executor multi with executor id 1246 parsl.dataflow.dflow INFO: Standard output for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/da2a8fa6-946e-48e0-8809-4576c1df689d_isrForCrosstalkSources_1031738_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/da2a8fa6-946e-48e0-8809-4576c1df689d_isrForCrosstalkSources_1031738_38.stderr parsl.dataflow.dflow DEBUG: Task 1246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1246 has memoization hash 780d0b50887a6cf7c1bb94e0e59048d6 parsl.dataflow.memoization INFO: Task 1246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1246 try 0 launched on executor multi with executor id 1247 parsl.dataflow.dflow INFO: Standard output for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/aa7b5dec-77be-4027-a3d9-3b28d3e1ee71_isrForCrosstalkSources_1031730_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/aa7b5dec-77be-4027-a3d9-3b28d3e1ee71_isrForCrosstalkSources_1031730_19.stderr parsl.dataflow.dflow DEBUG: Task 1247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1247 has memoization hash 169fa3fc0e235d2598e3c9df8d8d15d1 parsl.dataflow.memoization INFO: Task 1247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1247 try 0 launched on executor multi with executor id 1248 parsl.dataflow.dflow INFO: Standard output for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/50a48e88-c86b-4b31-bfa7-c3b775059636_isrForCrosstalkSources_1031701_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/50a48e88-c86b-4b31-bfa7-c3b775059636_isrForCrosstalkSources_1031701_19.stderr parsl.dataflow.dflow DEBUG: Task 1248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1248 has memoization hash d14306c92177df7e437bd640f7c5530f parsl.dataflow.memoization INFO: Task 1248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1248 try 0 launched on executor multi with executor id 1249 parsl.dataflow.dflow INFO: Standard output for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/3eaf4001-1db2-4856-9e98-0b2799d7cd18_isrForCrosstalkSources_1031702_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/3eaf4001-1db2-4856-9e98-0b2799d7cd18_isrForCrosstalkSources_1031702_1.stderr parsl.dataflow.dflow DEBUG: Task 1249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1249 has memoization hash 1b9b1cfc217f51823025553130e1c6fa parsl.dataflow.memoization INFO: Task 1249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1249 try 0 launched on executor multi with executor id 1250 parsl.dataflow.dflow INFO: Standard output for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f65b8ebc-313b-4ed9-ba4e-4d3260aef8d4_isrForCrosstalkSources_1031737_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f65b8ebc-313b-4ed9-ba4e-4d3260aef8d4_isrForCrosstalkSources_1031737_53.stderr parsl.dataflow.dflow DEBUG: Task 1250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1250 has memoization hash b0d4b0d2b0216df22b595610611210ae parsl.dataflow.memoization INFO: Task 1250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1250 try 0 launched on executor multi with executor id 1251 parsl.dataflow.dflow INFO: Standard output for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/be7d6871-e4bf-4d27-8dce-79fc643fa67a_isrForCrosstalkSources_1031737_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/be7d6871-e4bf-4d27-8dce-79fc643fa67a_isrForCrosstalkSources_1031737_26.stderr parsl.dataflow.dflow DEBUG: Task 1251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1251 has memoization hash 1be4269b1747669fdffc097172c55ab5 parsl.dataflow.memoization INFO: Task 1251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1251 try 0 launched on executor multi with executor id 1252 parsl.dataflow.dflow INFO: Standard output for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/45bcb014-1e27-4b60-ba39-6941bf2074d5_isrForCrosstalkSources_1031702_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/45bcb014-1e27-4b60-ba39-6941bf2074d5_isrForCrosstalkSources_1031702_61.stderr parsl.dataflow.dflow DEBUG: Task 1252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1252 has memoization hash d052f5068e06922c0c36904a49624b32 parsl.dataflow.memoization INFO: Task 1252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1252 try 0 launched on executor multi with executor id 1253 parsl.dataflow.dflow INFO: Standard output for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/dc046bde-3677-4281-aba1-9043a067dbaf_isrForCrosstalkSources_1031709_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/dc046bde-3677-4281-aba1-9043a067dbaf_isrForCrosstalkSources_1031709_35.stderr parsl.dataflow.dflow DEBUG: Task 1253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1253 has memoization hash 81728d7c640b69fab18ef9d680ac1073 parsl.dataflow.memoization INFO: Task 1253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1253 try 0 launched on executor multi with executor id 1254 parsl.dataflow.dflow INFO: Standard output for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/21b695ec-dc36-4bb9-b43f-ec7270461223_isrForCrosstalkSources_1031707_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/21b695ec-dc36-4bb9-b43f-ec7270461223_isrForCrosstalkSources_1031707_24.stderr parsl.dataflow.dflow DEBUG: Task 1254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1254 has memoization hash 75c3d281cd74d13086b15fe1b0ff85ae parsl.dataflow.memoization INFO: Task 1254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1254 try 0 launched on executor multi with executor id 1255 parsl.dataflow.dflow INFO: Standard output for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ffe3fa38-e8b4-4fbc-8e6d-b25fb9de11d6_isrForCrosstalkSources_1031737_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ffe3fa38-e8b4-4fbc-8e6d-b25fb9de11d6_isrForCrosstalkSources_1031737_52.stderr parsl.dataflow.dflow DEBUG: Task 1255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1255 has memoization hash ac3c16c55e8641db446809ccd775e456 parsl.dataflow.memoization INFO: Task 1255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1255 try 0 launched on executor multi with executor id 1256 parsl.dataflow.dflow INFO: Standard output for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/f1c454ae-e577-464d-a61d-2796dfdf5d53_isrForCrosstalkSources_1031734_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/f1c454ae-e577-464d-a61d-2796dfdf5d53_isrForCrosstalkSources_1031734_49.stderr parsl.dataflow.dflow DEBUG: Task 1256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1256 has memoization hash ccc1b1bfe0d62ae96704a347f3a7c7cd parsl.dataflow.memoization INFO: Task 1256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1256 try 0 launched on executor multi with executor id 1257 parsl.dataflow.dflow INFO: Standard output for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/542631e4-5288-473b-9b7e-528c8b2cd60c_isrForCrosstalkSources_1031734_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/542631e4-5288-473b-9b7e-528c8b2cd60c_isrForCrosstalkSources_1031734_62.stderr parsl.dataflow.dflow DEBUG: Task 1257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1257 has memoization hash 2f1040d6cafcc1e14f7f8bf29bf7e293 parsl.dataflow.memoization INFO: Task 1257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1257 try 0 launched on executor multi with executor id 1258 parsl.dataflow.dflow INFO: Standard output for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/708de2a3-9ed6-407a-b5cd-ccfa0e6dcc12_isrForCrosstalkSources_1031709_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/708de2a3-9ed6-407a-b5cd-ccfa0e6dcc12_isrForCrosstalkSources_1031709_28.stderr parsl.dataflow.dflow DEBUG: Task 1258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1258 has memoization hash 1d8021f54efc6ae9e8059b33675e611b parsl.dataflow.memoization INFO: Task 1258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1258 try 0 launched on executor multi with executor id 1259 parsl.dataflow.dflow INFO: Standard output for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/c2849103-1256-40c2-837b-fa1d779729c1_isrForCrosstalkSources_1031713_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/c2849103-1256-40c2-837b-fa1d779729c1_isrForCrosstalkSources_1031713_38.stderr parsl.dataflow.dflow DEBUG: Task 1259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1259 has memoization hash 9d97322e3672bce6594f9a7ad5c137e1 parsl.dataflow.memoization INFO: Task 1259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1259 try 0 launched on executor multi with executor id 1260 parsl.dataflow.dflow INFO: Standard output for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/4ec22b9a-9e81-4ac4-920a-4f3fa8b619d0_isrForCrosstalkSources_1031711_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/4ec22b9a-9e81-4ac4-920a-4f3fa8b619d0_isrForCrosstalkSources_1031711_60.stderr parsl.dataflow.dflow DEBUG: Task 1260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1260 has memoization hash 413d2e1bd95f949807bf4b8749b2e0f7 parsl.dataflow.memoization INFO: Task 1260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1260 try 0 launched on executor multi with executor id 1261 parsl.dataflow.dflow INFO: Standard output for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f651433c-de8e-40a7-b03c-8166a2108a8f_isrForCrosstalkSources_1031706_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f651433c-de8e-40a7-b03c-8166a2108a8f_isrForCrosstalkSources_1031706_25.stderr parsl.dataflow.dflow DEBUG: Task 1261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1261 has memoization hash 1c914e4e12afc8952b82b1a82ff61f18 parsl.dataflow.memoization INFO: Task 1261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb072e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1261 try 0 launched on executor multi with executor id 1262 parsl.dataflow.dflow INFO: Standard output for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d41dcbf0-f816-48ee-b26d-b51bed5a6863_isrForCrosstalkSources_1031701_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d41dcbf0-f816-48ee-b26d-b51bed5a6863_isrForCrosstalkSources_1031701_56.stderr parsl.dataflow.dflow DEBUG: Task 1262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1262 has memoization hash 174f5c8c39fd556ce702f6a9834dd454 parsl.dataflow.memoization INFO: Task 1262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1262 try 0 launched on executor multi with executor id 1263 parsl.dataflow.dflow INFO: Standard output for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5e9e9cde-1764-4ea4-b7ec-7451b26bbad5_isrForCrosstalkSources_1031714_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5e9e9cde-1764-4ea4-b7ec-7451b26bbad5_isrForCrosstalkSources_1031714_12.stderr parsl.dataflow.dflow DEBUG: Task 1263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1263 has memoization hash 1c87f9774f8c805e51279c5e9bc151fd parsl.dataflow.memoization INFO: Task 1263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1263 try 0 launched on executor multi with executor id 1264 parsl.dataflow.dflow INFO: Standard output for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/6465ecd8-efea-4bc5-ba64-ffe0ec3191ce_isrForCrosstalkSources_1031704_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/6465ecd8-efea-4bc5-ba64-ffe0ec3191ce_isrForCrosstalkSources_1031704_26.stderr parsl.dataflow.dflow DEBUG: Task 1264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1264 has memoization hash 5f8f48a87f8f15212dac3ff1130f3d6e parsl.dataflow.memoization INFO: Task 1264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1264 try 0 launched on executor multi with executor id 1265 parsl.dataflow.dflow INFO: Standard output for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5a3a9a8e-6bf5-4d49-b556-275d9389a5b1_isrForCrosstalkSources_1031728_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5a3a9a8e-6bf5-4d49-b556-275d9389a5b1_isrForCrosstalkSources_1031728_35.stderr parsl.dataflow.dflow DEBUG: Task 1265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1265 has memoization hash 11ad87d8d1df226f34da63f948e88e9f parsl.dataflow.memoization INFO: Task 1265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1265 try 0 launched on executor multi with executor id 1266 parsl.dataflow.dflow INFO: Standard output for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a20f1158-e531-4105-b154-7d6de93767bf_isrForCrosstalkSources_1031718_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a20f1158-e531-4105-b154-7d6de93767bf_isrForCrosstalkSources_1031718_19.stderr parsl.dataflow.dflow DEBUG: Task 1266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1266 has memoization hash d07e4e22cae680e38b43214e422334aa parsl.dataflow.memoization INFO: Task 1266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1266 try 0 launched on executor multi with executor id 1267 parsl.dataflow.dflow INFO: Standard output for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/aff86859-62f6-418a-b541-4bfdf1cf2393_isrForCrosstalkSources_1031718_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/aff86859-62f6-418a-b541-4bfdf1cf2393_isrForCrosstalkSources_1031718_13.stderr parsl.dataflow.dflow DEBUG: Task 1267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1267 has memoization hash c5dfbdf769bec2f6a6061db9718ef88c parsl.dataflow.memoization INFO: Task 1267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb074c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1267 try 0 launched on executor multi with executor id 1268 parsl.dataflow.dflow INFO: Standard output for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/4fd6c007-d2a4-4c1f-a6f7-0ba7d699ef0e_isrForCrosstalkSources_1031735_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/4fd6c007-d2a4-4c1f-a6f7-0ba7d699ef0e_isrForCrosstalkSources_1031735_59.stderr parsl.dataflow.dflow DEBUG: Task 1268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1268 has memoization hash 702441d3c3280df77af565b6ebf6d85d parsl.dataflow.memoization INFO: Task 1268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1268 try 0 launched on executor multi with executor id 1269 parsl.dataflow.dflow INFO: Standard output for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/fefae2f9-980f-450c-ae66-0098153ea77b_isrForCrosstalkSources_1031716_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/fefae2f9-980f-450c-ae66-0098153ea77b_isrForCrosstalkSources_1031716_29.stderr parsl.dataflow.dflow DEBUG: Task 1269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1269 has memoization hash 25e79368e6dac1e78862d851a6bad60b parsl.dataflow.memoization INFO: Task 1269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1269 try 0 launched on executor multi with executor id 1270 parsl.dataflow.dflow INFO: Standard output for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e91f427c-0891-4a19-b2ce-90bec49b5f1c_isrForCrosstalkSources_1031707_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e91f427c-0891-4a19-b2ce-90bec49b5f1c_isrForCrosstalkSources_1031707_38.stderr parsl.dataflow.dflow DEBUG: Task 1270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1270 has memoization hash 5c83d67337f3ce065ccbebfbbd6f42ac parsl.dataflow.memoization INFO: Task 1270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1270 try 0 launched on executor multi with executor id 1271 parsl.dataflow.dflow INFO: Standard output for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/cb724ebe-85c7-491a-84dc-9eed44c4e720_isrForCrosstalkSources_1031723_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/cb724ebe-85c7-491a-84dc-9eed44c4e720_isrForCrosstalkSources_1031723_59.stderr parsl.dataflow.dflow DEBUG: Task 1271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1271 has memoization hash 789df8e6f97981cd697952e82bd9a497 parsl.dataflow.memoization INFO: Task 1271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1271 try 0 launched on executor multi with executor id 1272 parsl.dataflow.dflow INFO: Standard output for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/41d25e2e-b09d-4f2f-9441-b3052a9d769c_isrForCrosstalkSources_1031721_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/41d25e2e-b09d-4f2f-9441-b3052a9d769c_isrForCrosstalkSources_1031721_58.stderr parsl.dataflow.dflow DEBUG: Task 1272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1272 has memoization hash f481617a735f4448a8773100c33afb66 parsl.dataflow.memoization INFO: Task 1272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1272 try 0 launched on executor multi with executor id 1273 parsl.dataflow.dflow INFO: Standard output for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/402e6c20-a9d4-481e-bd15-24dd319e2f27_isrForCrosstalkSources_1031728_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/402e6c20-a9d4-481e-bd15-24dd319e2f27_isrForCrosstalkSources_1031728_24.stderr parsl.dataflow.dflow DEBUG: Task 1273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1273 has memoization hash 7d674dd54a36f64ba0239999e32f9041 parsl.dataflow.memoization INFO: Task 1273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1273 try 0 launched on executor multi with executor id 1274 parsl.dataflow.dflow INFO: Standard output for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/d770c088-e78c-43f9-be66-211538a53029_isrForCrosstalkSources_1031712_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/d770c088-e78c-43f9-be66-211538a53029_isrForCrosstalkSources_1031712_28.stderr parsl.dataflow.dflow DEBUG: Task 1274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1274 has memoization hash 5c89082e500297780f2cceacab194c46 parsl.dataflow.memoization INFO: Task 1274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1274 try 0 launched on executor multi with executor id 1275 parsl.dataflow.dflow INFO: Standard output for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/24e1abc0-832f-42ff-b576-1fb70ffbf828_isrForCrosstalkSources_1031713_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/24e1abc0-832f-42ff-b576-1fb70ffbf828_isrForCrosstalkSources_1031713_22.stderr parsl.dataflow.dflow DEBUG: Task 1275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1275 has memoization hash b122db7eea856201896305bd7e980fb5 parsl.dataflow.memoization INFO: Task 1275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1275 try 0 launched on executor multi with executor id 1276 parsl.dataflow.dflow INFO: Standard output for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/310de23e-295c-48d1-879a-12369642499b_isrForCrosstalkSources_1031714_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/310de23e-295c-48d1-879a-12369642499b_isrForCrosstalkSources_1031714_4.stderr parsl.dataflow.dflow DEBUG: Task 1276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1276 has memoization hash 4560a8fb8abcc4a325cbe107f4965bad parsl.dataflow.memoization INFO: Task 1276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1276 try 0 launched on executor multi with executor id 1277 parsl.dataflow.dflow INFO: Standard output for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d39c8fb9-b275-4fb3-81c0-42a15f9cd265_isrForCrosstalkSources_1031709_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d39c8fb9-b275-4fb3-81c0-42a15f9cd265_isrForCrosstalkSources_1031709_53.stderr parsl.dataflow.dflow DEBUG: Task 1277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1277 has memoization hash f7fb72d6d14e0940ce60edfd9ee49389 parsl.dataflow.memoization INFO: Task 1277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1277 try 0 launched on executor multi with executor id 1278 parsl.dataflow.dflow INFO: Standard output for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/9205921a-2b8b-4dfb-90ad-2a0d967d8035_isrForCrosstalkSources_1031705_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/9205921a-2b8b-4dfb-90ad-2a0d967d8035_isrForCrosstalkSources_1031705_26.stderr parsl.dataflow.dflow DEBUG: Task 1278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1278 has memoization hash 951a0d265475a42c87bfaa733808486a parsl.dataflow.memoization INFO: Task 1278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1278 try 0 launched on executor multi with executor id 1279 parsl.dataflow.dflow INFO: Standard output for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1370879e-5d01-47b8-95ec-1f8e9e6b926e_isrForCrosstalkSources_1031715_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1370879e-5d01-47b8-95ec-1f8e9e6b926e_isrForCrosstalkSources_1031715_15.stderr parsl.dataflow.dflow DEBUG: Task 1279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1279 has memoization hash 82a9019d045e3f9c875fac124a8e4458 parsl.dataflow.memoization INFO: Task 1279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1279 try 0 launched on executor multi with executor id 1280 parsl.dataflow.dflow INFO: Standard output for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/7e3ace4c-c10b-408a-b345-0a04c6680bc9_isrForCrosstalkSources_1031709_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/7e3ace4c-c10b-408a-b345-0a04c6680bc9_isrForCrosstalkSources_1031709_31.stderr parsl.dataflow.dflow DEBUG: Task 1280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1280 has memoization hash ec554171ed8935a920165b23c978c7a2 parsl.dataflow.memoization INFO: Task 1280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1280 try 0 launched on executor multi with executor id 1281 parsl.dataflow.dflow INFO: Standard output for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4fd92a0c-3a81-4a29-9596-0468aad29ee0_isrForCrosstalkSources_1031733_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4fd92a0c-3a81-4a29-9596-0468aad29ee0_isrForCrosstalkSources_1031733_36.stderr parsl.dataflow.dflow DEBUG: Task 1281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1281 has memoization hash bfbe8199e7805575a9d1ec972199eb7e parsl.dataflow.memoization INFO: Task 1281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1281 try 0 launched on executor multi with executor id 1282 parsl.dataflow.dflow INFO: Standard output for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4abf6fbb-c52b-46e0-a53f-a7b77eda992a_isrForCrosstalkSources_1031703_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4abf6fbb-c52b-46e0-a53f-a7b77eda992a_isrForCrosstalkSources_1031703_47.stderr parsl.dataflow.dflow DEBUG: Task 1282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1282 has memoization hash 5a9fdcd87a4ae226ffe1e521d7204bf5 parsl.dataflow.memoization INFO: Task 1282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1282 try 0 launched on executor multi with executor id 1283 parsl.dataflow.dflow INFO: Standard output for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/4f90df70-0129-42fe-82a5-d1eabe9b983b_isrForCrosstalkSources_1031736_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/4f90df70-0129-42fe-82a5-d1eabe9b983b_isrForCrosstalkSources_1031736_59.stderr parsl.dataflow.dflow DEBUG: Task 1283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1283 has memoization hash 632dffb5a91258e8aabac32ebb0b7de0 parsl.dataflow.memoization INFO: Task 1283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1283 try 0 launched on executor multi with executor id 1284 parsl.dataflow.dflow INFO: Standard output for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/bd63992f-2e1e-42d5-88b8-3a4300425968_isrForCrosstalkSources_1031719_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/bd63992f-2e1e-42d5-88b8-3a4300425968_isrForCrosstalkSources_1031719_44.stderr parsl.dataflow.dflow DEBUG: Task 1284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1284 has memoization hash 222b0d8c9efd75d3fbaf681512bbad8b parsl.dataflow.memoization INFO: Task 1284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1284 try 0 launched on executor multi with executor id 1285 parsl.dataflow.dflow INFO: Standard output for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6de02fcb-493b-4409-a687-42c311124fdd_isrForCrosstalkSources_1031716_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6de02fcb-493b-4409-a687-42c311124fdd_isrForCrosstalkSources_1031716_9.stderr parsl.dataflow.dflow DEBUG: Task 1285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1285 has memoization hash 0499fa23aa41254a5b9436748a2cad7a parsl.dataflow.memoization INFO: Task 1285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1285 try 0 launched on executor multi with executor id 1286 parsl.dataflow.dflow INFO: Standard output for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a4136a3d-7a9d-4f8f-bc72-e872c1e933a3_isrForCrosstalkSources_1031722_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a4136a3d-7a9d-4f8f-bc72-e872c1e933a3_isrForCrosstalkSources_1031722_42.stderr parsl.dataflow.dflow DEBUG: Task 1286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1286 has memoization hash 102ea08450c6a21c4991cbcf9a020bf5 parsl.dataflow.memoization INFO: Task 1286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1286 try 0 launched on executor multi with executor id 1287 parsl.dataflow.dflow INFO: Standard output for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/7962f207-73c9-485d-b498-edcf4105cd3e_isrForCrosstalkSources_1031703_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/7962f207-73c9-485d-b498-edcf4105cd3e_isrForCrosstalkSources_1031703_12.stderr parsl.dataflow.dflow DEBUG: Task 1287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1287 has memoization hash 343684330d79a073a6c68f7a7cc3fd05 parsl.dataflow.memoization INFO: Task 1287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1287 try 0 launched on executor multi with executor id 1288 parsl.dataflow.dflow INFO: Standard output for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/ce7256f0-2463-42d6-aaf1-b9b189af396b_isrForCrosstalkSources_1031703_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/ce7256f0-2463-42d6-aaf1-b9b189af396b_isrForCrosstalkSources_1031703_31.stderr parsl.dataflow.dflow DEBUG: Task 1288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1288 has memoization hash 4a32be9c95f58e549acaa36e90ac56e8 parsl.dataflow.memoization INFO: Task 1288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1288 try 0 launched on executor multi with executor id 1289 parsl.dataflow.dflow INFO: Standard output for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0931d5af-cff1-43fa-be80-06988d8d507f_isrForCrosstalkSources_1031737_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0931d5af-cff1-43fa-be80-06988d8d507f_isrForCrosstalkSources_1031737_60.stderr parsl.dataflow.dflow DEBUG: Task 1289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1289 has memoization hash 67a213cc02fc4883f8af343807846c87 parsl.dataflow.memoization INFO: Task 1289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1289 try 0 launched on executor multi with executor id 1290 parsl.dataflow.dflow INFO: Standard output for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5a00368c-db7c-4e62-96ba-1e02fbcb1d2c_isrForCrosstalkSources_1031728_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5a00368c-db7c-4e62-96ba-1e02fbcb1d2c_isrForCrosstalkSources_1031728_47.stderr parsl.dataflow.dflow DEBUG: Task 1290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1290 has memoization hash cc5c43c26cf3f905f6faf4b954489e30 parsl.dataflow.memoization INFO: Task 1290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1290 try 0 launched on executor multi with executor id 1291 parsl.dataflow.dflow INFO: Standard output for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/353cb1fc-50b7-4015-88bf-ff9feb222fa9_isrForCrosstalkSources_1031704_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/353cb1fc-50b7-4015-88bf-ff9feb222fa9_isrForCrosstalkSources_1031704_11.stderr parsl.dataflow.dflow DEBUG: Task 1291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1291 has memoization hash bf7833a104e901e0ae5679160369362d parsl.dataflow.memoization INFO: Task 1291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1291 try 0 launched on executor multi with executor id 1292 parsl.dataflow.dflow INFO: Standard output for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f2ab1b01-0216-49eb-965f-fbd71be200ce_isrForCrosstalkSources_1031727_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f2ab1b01-0216-49eb-965f-fbd71be200ce_isrForCrosstalkSources_1031727_53.stderr parsl.dataflow.dflow DEBUG: Task 1292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1292 has memoization hash 7edeacc6e903b4c0553b28e8661d81b8 parsl.dataflow.memoization INFO: Task 1292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1292 try 0 launched on executor multi with executor id 1293 parsl.dataflow.dflow INFO: Standard output for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d83f2e92-1e57-4607-b673-b3b6e6565ff5_isrForCrosstalkSources_1031711_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d83f2e92-1e57-4607-b673-b3b6e6565ff5_isrForCrosstalkSources_1031711_14.stderr parsl.dataflow.dflow DEBUG: Task 1293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1293 has memoization hash 220b7e5a06c1dd662304a19380567a0e parsl.dataflow.memoization INFO: Task 1293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1293 try 0 launched on executor multi with executor id 1294 parsl.dataflow.dflow INFO: Standard output for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/9f7980f9-894c-4b72-ac8e-bb4e3fa154af_isrForCrosstalkSources_1031710_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/9f7980f9-894c-4b72-ac8e-bb4e3fa154af_isrForCrosstalkSources_1031710_30.stderr parsl.dataflow.dflow DEBUG: Task 1294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1294 has memoization hash 0e3820610fc889001309b2a742b4e392 parsl.dataflow.memoization INFO: Task 1294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:12 proc_lsst.multi:170 [INFO] [multi] job_id 17147948 proc_lsst.multi INFO: [multi] job_id 17147948 2024-03-17 04:02:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-17 04:02:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147948 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147948 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17147948 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-17 04:02:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-17 04:02:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-17 04:02:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1294 try 0 launched on executor multi with executor id 1295 parsl.dataflow.dflow INFO: Standard output for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/6e79e23c-454f-4b70-af05-b954b3fb34cb_isrForCrosstalkSources_1031717_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/6e79e23c-454f-4b70-af05-b954b3fb34cb_isrForCrosstalkSources_1031717_46.stderr parsl.dataflow.dflow DEBUG: Task 1295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1295 has memoization hash f8f8e30a88d3e38a34d687037d257a5a parsl.dataflow.memoization INFO: Task 1295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb07f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1295 try 0 launched on executor multi with executor id 1296 parsl.dataflow.dflow INFO: Standard output for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/b0f61227-9ba6-4304-9887-2480b5aab81f_isrForCrosstalkSources_1031720_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/b0f61227-9ba6-4304-9887-2480b5aab81f_isrForCrosstalkSources_1031720_26.stderr parsl.dataflow.dflow DEBUG: Task 1296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1296 has memoization hash 33dca735c29bdc31bdcad223ecafab49 parsl.dataflow.memoization INFO: Task 1296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1296 try 0 launched on executor multi with executor id 1297 parsl.dataflow.dflow INFO: Standard output for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/e03b3abf-3b56-489d-86b6-5fd7bf466b4d_isrForCrosstalkSources_1031713_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/e03b3abf-3b56-489d-86b6-5fd7bf466b4d_isrForCrosstalkSources_1031713_4.stderr parsl.dataflow.dflow DEBUG: Task 1297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1297 has memoization hash e7b72ef44442daa503b2ac0d191d6a26 parsl.dataflow.memoization INFO: Task 1297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb06980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1297 try 0 launched on executor multi with executor id 1298 parsl.dataflow.dflow INFO: Standard output for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0518d1ca-20e0-4f7d-9dec-44979220548a_isrForCrosstalkSources_1031717_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0518d1ca-20e0-4f7d-9dec-44979220548a_isrForCrosstalkSources_1031717_43.stderr parsl.dataflow.dflow DEBUG: Task 1298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1298 has memoization hash 3de01af2c05e3bf0e5b08f5584d9440f parsl.dataflow.memoization INFO: Task 1298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1298 try 0 launched on executor multi with executor id 1299 parsl.dataflow.dflow INFO: Standard output for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/f846e33a-53ea-4468-86e9-51a5306cacfb_isrForCrosstalkSources_1031722_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/f846e33a-53ea-4468-86e9-51a5306cacfb_isrForCrosstalkSources_1031722_8.stderr parsl.dataflow.dflow DEBUG: Task 1299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1299 has memoization hash a3e40fc9ed7418de22e4d0f2b640e8bb parsl.dataflow.memoization INFO: Task 1299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1299 try 0 launched on executor multi with executor id 1300 parsl.dataflow.dflow INFO: Standard output for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/551030c7-1d4a-478c-a1d2-c12089dc1406_isrForCrosstalkSources_1031717_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/551030c7-1d4a-478c-a1d2-c12089dc1406_isrForCrosstalkSources_1031717_10.stderr parsl.dataflow.dflow DEBUG: Task 1300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1300 has memoization hash 9f1961ee4b71047c48ff748ce327025b parsl.dataflow.memoization INFO: Task 1300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1300 try 0 launched on executor multi with executor id 1301 parsl.dataflow.dflow INFO: Standard output for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/16ecc51d-7d31-41c9-b49a-2a4268af422e_isrForCrosstalkSources_1031735_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/16ecc51d-7d31-41c9-b49a-2a4268af422e_isrForCrosstalkSources_1031735_55.stderr parsl.dataflow.dflow DEBUG: Task 1301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1301 has memoization hash ab01237b34d3d1b814dae56a6b63cd50 parsl.dataflow.memoization INFO: Task 1301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1301 try 0 launched on executor multi with executor id 1302 parsl.dataflow.dflow INFO: Standard output for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d5e061a9-9820-40cd-876b-db8d086a29e4_isrForCrosstalkSources_1031715_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d5e061a9-9820-40cd-876b-db8d086a29e4_isrForCrosstalkSources_1031715_20.stderr parsl.dataflow.dflow DEBUG: Task 1302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1302 has memoization hash d084f29361feae1472ddd82752e1d9e3 parsl.dataflow.memoization INFO: Task 1302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1302 try 0 launched on executor multi with executor id 1303 parsl.dataflow.dflow INFO: Standard output for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c55bf889-c50a-4600-8378-17dc79a56053_isrForCrosstalkSources_1031722_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c55bf889-c50a-4600-8378-17dc79a56053_isrForCrosstalkSources_1031722_34.stderr parsl.dataflow.dflow DEBUG: Task 1303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1303 has memoization hash 93b816a6efcffdbf3a901871560db21e parsl.dataflow.memoization INFO: Task 1303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1303 try 0 launched on executor multi with executor id 1304 parsl.dataflow.dflow INFO: Standard output for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f6975a89-7b68-4f3b-b748-ab5c1ade366e_isrForCrosstalkSources_1031730_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f6975a89-7b68-4f3b-b748-ab5c1ade366e_isrForCrosstalkSources_1031730_26.stderr parsl.dataflow.dflow DEBUG: Task 1304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1304 has memoization hash 212e5b7cb3bd46c93ebcc5b5c28c117c parsl.dataflow.memoization INFO: Task 1304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1304 try 0 launched on executor multi with executor id 1305 parsl.dataflow.dflow INFO: Standard output for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/1db45182-d5d4-4baa-aa1a-b7be52c7d330_isrForCrosstalkSources_1031730_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/1db45182-d5d4-4baa-aa1a-b7be52c7d330_isrForCrosstalkSources_1031730_7.stderr parsl.dataflow.dflow DEBUG: Task 1305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1305 has memoization hash 45740c40360a36d0c46caef6453bfbdc parsl.dataflow.memoization INFO: Task 1305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1305 try 0 launched on executor multi with executor id 1306 parsl.dataflow.dflow INFO: Standard output for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8b46b0a8-37bf-4ec4-9a68-a8b8cfdc9a45_isrForCrosstalkSources_1031719_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8b46b0a8-37bf-4ec4-9a68-a8b8cfdc9a45_isrForCrosstalkSources_1031719_29.stderr parsl.dataflow.dflow DEBUG: Task 1306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1306 has memoization hash 2ff1be5557383fff09db8b4a842fe95e parsl.dataflow.memoization INFO: Task 1306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1306 try 0 launched on executor multi with executor id 1307 parsl.dataflow.dflow INFO: Standard output for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b7ff2e10-363d-4c39-bdad-ce19a0584e97_isrForCrosstalkSources_1031718_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b7ff2e10-363d-4c39-bdad-ce19a0584e97_isrForCrosstalkSources_1031718_14.stderr parsl.dataflow.dflow DEBUG: Task 1307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1307 has memoization hash 05a9e93e5f9113fc79279a955bdaec2e parsl.dataflow.memoization INFO: Task 1307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1307 try 0 launched on executor multi with executor id 1308 parsl.dataflow.dflow INFO: Standard output for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1fb6ef36-ec72-48d1-a258-1133363d89fd_isrForCrosstalkSources_1031703_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1fb6ef36-ec72-48d1-a258-1133363d89fd_isrForCrosstalkSources_1031703_34.stderr parsl.dataflow.dflow DEBUG: Task 1308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1308 has memoization hash 811ea236b91678c942a84cb0c4fa7a90 parsl.dataflow.memoization INFO: Task 1308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1308 try 0 launched on executor multi with executor id 1309 parsl.dataflow.dflow INFO: Standard output for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a70254c0-12cc-4840-8231-d7b46efbce68_isrForCrosstalkSources_1031725_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a70254c0-12cc-4840-8231-d7b46efbce68_isrForCrosstalkSources_1031725_30.stderr parsl.dataflow.dflow DEBUG: Task 1309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1309 has memoization hash ba7ac023010c393e19c4c423321c3829 parsl.dataflow.memoization INFO: Task 1309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1309 try 0 launched on executor multi with executor id 1310 parsl.dataflow.dflow INFO: Standard output for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/dedab0be-b843-4cbc-abfb-c4374b31ca7f_isrForCrosstalkSources_1031700_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/dedab0be-b843-4cbc-abfb-c4374b31ca7f_isrForCrosstalkSources_1031700_1.stderr parsl.dataflow.dflow DEBUG: Task 1310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1310 has memoization hash ee2bcf5a33ced60c7597ac6574f50727 parsl.dataflow.memoization INFO: Task 1310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1310 try 0 launched on executor multi with executor id 1311 parsl.dataflow.dflow INFO: Standard output for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1e81bb80-71a5-43f5-8a17-4aab894630f3_isrForCrosstalkSources_1031701_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1e81bb80-71a5-43f5-8a17-4aab894630f3_isrForCrosstalkSources_1031701_60.stderr parsl.dataflow.dflow DEBUG: Task 1311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1311 has memoization hash 5de8a90c96f2b370ab8874cd29b4f39f parsl.dataflow.memoization INFO: Task 1311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1311 try 0 launched on executor multi with executor id 1312 parsl.dataflow.dflow INFO: Standard output for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b18f8939-5c2f-4d5a-8801-f963076fa5a6_isrForCrosstalkSources_1031714_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b18f8939-5c2f-4d5a-8801-f963076fa5a6_isrForCrosstalkSources_1031714_7.stderr parsl.dataflow.dflow DEBUG: Task 1312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1312 has memoization hash 50d89f388b816210d8ada114fd379621 parsl.dataflow.memoization INFO: Task 1312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1312 try 0 launched on executor multi with executor id 1313 parsl.dataflow.dflow INFO: Standard output for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/21c65279-b20a-4d9a-a14b-09ccb39643cf_isrForCrosstalkSources_1031726_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/21c65279-b20a-4d9a-a14b-09ccb39643cf_isrForCrosstalkSources_1031726_10.stderr parsl.dataflow.dflow DEBUG: Task 1313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1313 has memoization hash 0cc1dafb1f26b0e1cf1377793c9f2b49 parsl.dataflow.memoization INFO: Task 1313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1313 try 0 launched on executor multi with executor id 1314 parsl.dataflow.dflow INFO: Standard output for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/6a23b99e-68fe-49fe-abda-234e11ef609f_isrForCrosstalkSources_1031735_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/6a23b99e-68fe-49fe-abda-234e11ef609f_isrForCrosstalkSources_1031735_47.stderr parsl.dataflow.dflow DEBUG: Task 1314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1314 has memoization hash 090868f1dd5d12baf3e5daaed25e60e6 parsl.dataflow.memoization INFO: Task 1314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1314 try 0 launched on executor multi with executor id 1315 parsl.dataflow.dflow INFO: Standard output for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/666ae8a8-329b-40fc-9d50-08e365aae8b0_isrForCrosstalkSources_1031721_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/666ae8a8-329b-40fc-9d50-08e365aae8b0_isrForCrosstalkSources_1031721_15.stderr parsl.dataflow.dflow DEBUG: Task 1315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1315 has memoization hash 54df400acdbc4b438b2c8b02a14b530f parsl.dataflow.memoization INFO: Task 1315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1315 try 0 launched on executor multi with executor id 1316 parsl.dataflow.dflow INFO: Standard output for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7a4d1660-9fed-49fa-9c01-537f2cb600be_isrForCrosstalkSources_1031701_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7a4d1660-9fed-49fa-9c01-537f2cb600be_isrForCrosstalkSources_1031701_16.stderr parsl.dataflow.dflow DEBUG: Task 1316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1316 has memoization hash be1da325039898b1082392f4fb2382db parsl.dataflow.memoization INFO: Task 1316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1316 try 0 launched on executor multi with executor id 1317 parsl.dataflow.dflow INFO: Standard output for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/31f76ab4-b32e-4c90-b717-48d58d842c24_isrForCrosstalkSources_1031701_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/31f76ab4-b32e-4c90-b717-48d58d842c24_isrForCrosstalkSources_1031701_42.stderr parsl.dataflow.dflow DEBUG: Task 1317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1317 has memoization hash 1103236a98fe2448eb2e6335957d2f44 parsl.dataflow.memoization INFO: Task 1317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1317 try 0 launched on executor multi with executor id 1318 parsl.dataflow.dflow INFO: Standard output for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/2f76929d-af22-42da-852d-e55341c159da_isrForCrosstalkSources_1031701_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/2f76929d-af22-42da-852d-e55341c159da_isrForCrosstalkSources_1031701_50.stderr parsl.dataflow.dflow DEBUG: Task 1318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1318 has memoization hash e89a51361f7c26af200d61ed65508a7a parsl.dataflow.memoization INFO: Task 1318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1318 try 0 launched on executor multi with executor id 1319 parsl.dataflow.dflow INFO: Standard output for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0222a112-71e1-40bb-aa7b-d658cdbc2adf_isrForCrosstalkSources_1031705_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0222a112-71e1-40bb-aa7b-d658cdbc2adf_isrForCrosstalkSources_1031705_33.stderr parsl.dataflow.dflow DEBUG: Task 1319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1319 has memoization hash 11c229444516f93bff858d8d07529eed parsl.dataflow.memoization INFO: Task 1319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1319 try 0 launched on executor multi with executor id 1320 parsl.dataflow.dflow INFO: Standard output for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/e198b303-35b6-4dee-9e5a-66105e08a110_isrForCrosstalkSources_1031702_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/e198b303-35b6-4dee-9e5a-66105e08a110_isrForCrosstalkSources_1031702_43.stderr parsl.dataflow.dflow DEBUG: Task 1320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1320 has memoization hash e899c9361fa32992c52f77ba67658f87 parsl.dataflow.memoization INFO: Task 1320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1320 try 0 launched on executor multi with executor id 1321 parsl.dataflow.dflow INFO: Standard output for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/15100d57-cc45-4cbf-aa8d-204ed17caaba_isrForCrosstalkSources_1031727_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/15100d57-cc45-4cbf-aa8d-204ed17caaba_isrForCrosstalkSources_1031727_13.stderr parsl.dataflow.dflow DEBUG: Task 1321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1321 has memoization hash 6139bb2e98ebb2335be57cb8eb5781c4 parsl.dataflow.memoization INFO: Task 1321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1321 try 0 launched on executor multi with executor id 1322 parsl.dataflow.dflow INFO: Standard output for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b4d8deb8-c8cb-45a3-8d29-89635ad20383_isrForCrosstalkSources_1031703_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b4d8deb8-c8cb-45a3-8d29-89635ad20383_isrForCrosstalkSources_1031703_39.stderr parsl.dataflow.dflow DEBUG: Task 1322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1322 has memoization hash ad56c459e10d10e308fad83caa548bfd parsl.dataflow.memoization INFO: Task 1322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1322 try 0 launched on executor multi with executor id 1323 parsl.dataflow.dflow INFO: Standard output for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/b4db0d1f-e53c-4430-81ad-a4bf916f4f1e_isrForCrosstalkSources_1031720_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/b4db0d1f-e53c-4430-81ad-a4bf916f4f1e_isrForCrosstalkSources_1031720_18.stderr parsl.dataflow.dflow DEBUG: Task 1323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1323 has memoization hash 47a809405525b7d6cb707cd91a07805e parsl.dataflow.memoization INFO: Task 1323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1323 try 0 launched on executor multi with executor id 1324 parsl.dataflow.dflow INFO: Standard output for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d00c3c69-3697-42b4-b6c0-24bad7053af8_isrForCrosstalkSources_1031733_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d00c3c69-3697-42b4-b6c0-24bad7053af8_isrForCrosstalkSources_1031733_21.stderr parsl.dataflow.dflow DEBUG: Task 1324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1324 has memoization hash 9d173c178645ca7bd9e7c15ae1ae185d parsl.dataflow.memoization INFO: Task 1324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1324 try 0 launched on executor multi with executor id 1325 parsl.dataflow.dflow INFO: Standard output for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8e8c7867-6faf-451c-a32b-ecb127fb07d5_isrForCrosstalkSources_1031708_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8e8c7867-6faf-451c-a32b-ecb127fb07d5_isrForCrosstalkSources_1031708_28.stderr parsl.dataflow.dflow DEBUG: Task 1325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1325 has memoization hash 40992000554dcebfe6c3499b9d18d424 parsl.dataflow.memoization INFO: Task 1325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1325 try 0 launched on executor multi with executor id 1326 parsl.dataflow.dflow INFO: Standard output for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5b2d977f-9c17-4961-9c4a-ca67af2485bc_isrForCrosstalkSources_1031717_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5b2d977f-9c17-4961-9c4a-ca67af2485bc_isrForCrosstalkSources_1031717_59.stderr parsl.dataflow.dflow DEBUG: Task 1326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1326 has memoization hash d9983b522d48805b2f8ec1141ece3781 parsl.dataflow.memoization INFO: Task 1326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1326 try 0 launched on executor multi with executor id 1327 parsl.dataflow.dflow INFO: Standard output for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a117298f-2828-4c7d-bfcb-9c47a29fcccd_isrForCrosstalkSources_1031711_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a117298f-2828-4c7d-bfcb-9c47a29fcccd_isrForCrosstalkSources_1031711_43.stderr parsl.dataflow.dflow DEBUG: Task 1327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1327 has memoization hash a69057ef037415c84a6483eecd6e0f30 parsl.dataflow.memoization INFO: Task 1327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1327 try 0 launched on executor multi with executor id 1328 parsl.dataflow.dflow INFO: Standard output for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/abcd5295-2a17-46aa-9f1a-15504cc6db66_isrForCrosstalkSources_1031727_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/abcd5295-2a17-46aa-9f1a-15504cc6db66_isrForCrosstalkSources_1031727_57.stderr parsl.dataflow.dflow DEBUG: Task 1328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1328 has memoization hash adfd46425d4484b84f78e3902572ec0d parsl.dataflow.memoization INFO: Task 1328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1328 try 0 launched on executor multi with executor id 1329 parsl.dataflow.dflow INFO: Standard output for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/4a323272-dbc0-4cbb-a201-623b539beaf9_isrForCrosstalkSources_1031700_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/4a323272-dbc0-4cbb-a201-623b539beaf9_isrForCrosstalkSources_1031700_59.stderr parsl.dataflow.dflow DEBUG: Task 1329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1329 has memoization hash a0dd1af97ac668e7206e1e61d9bd00b6 parsl.dataflow.memoization INFO: Task 1329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1329 try 0 launched on executor multi with executor id 1330 parsl.dataflow.dflow INFO: Standard output for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/3d6b7c61-605e-47f0-91b4-16620cecd163_isrForCrosstalkSources_1031734_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/3d6b7c61-605e-47f0-91b4-16620cecd163_isrForCrosstalkSources_1031734_59.stderr parsl.dataflow.dflow DEBUG: Task 1330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1330 has memoization hash 95f6de0c3f9b99e4b01a5ccce577f3f3 parsl.dataflow.memoization INFO: Task 1330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1330 try 0 launched on executor multi with executor id 1331 parsl.dataflow.dflow INFO: Standard output for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/24c5c019-3fa8-4769-b871-4dd5247155a2_isrForCrosstalkSources_1031716_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/24c5c019-3fa8-4769-b871-4dd5247155a2_isrForCrosstalkSources_1031716_48.stderr parsl.dataflow.dflow DEBUG: Task 1331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1331 has memoization hash 2db1f73adcefa3ec6f5969f95793634a parsl.dataflow.memoization INFO: Task 1331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1331 try 0 launched on executor multi with executor id 1332 parsl.dataflow.dflow INFO: Standard output for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e0b2580e-aa4f-4f7a-b51d-a217ec1ab4ef_isrForCrosstalkSources_1031725_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e0b2580e-aa4f-4f7a-b51d-a217ec1ab4ef_isrForCrosstalkSources_1031725_9.stderr parsl.dataflow.dflow DEBUG: Task 1332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1332 has memoization hash 8dc6c02979e7e29a06887f1d122a2dd1 parsl.dataflow.memoization INFO: Task 1332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1332 try 0 launched on executor multi with executor id 1333 parsl.dataflow.dflow INFO: Standard output for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/7449f93d-b28d-4281-88f6-77235a799cd5_isrForCrosstalkSources_1031734_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/7449f93d-b28d-4281-88f6-77235a799cd5_isrForCrosstalkSources_1031734_43.stderr parsl.dataflow.dflow DEBUG: Task 1333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1333 has memoization hash fa980d427e3e3cba33cd6688e2f5e2ec parsl.dataflow.memoization INFO: Task 1333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1333 try 0 launched on executor multi with executor id 1334 parsl.dataflow.dflow INFO: Standard output for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a5e66230-4930-4262-8cdd-3afce305ddc9_isrForCrosstalkSources_1031713_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a5e66230-4930-4262-8cdd-3afce305ddc9_isrForCrosstalkSources_1031713_2.stderr parsl.dataflow.dflow DEBUG: Task 1334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1334 has memoization hash 778424396274e60b6ba321c7f8b88626 parsl.dataflow.memoization INFO: Task 1334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1334 try 0 launched on executor multi with executor id 1335 parsl.dataflow.dflow INFO: Standard output for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0ad7cb09-c396-4989-b738-302ead4344b6_isrForCrosstalkSources_1031724_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0ad7cb09-c396-4989-b738-302ead4344b6_isrForCrosstalkSources_1031724_56.stderr parsl.dataflow.dflow DEBUG: Task 1335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1335 has memoization hash db7a5e3a16bcef90ac2ab10c3b9dcdd6 parsl.dataflow.memoization INFO: Task 1335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1335 try 0 launched on executor multi with executor id 1336 parsl.dataflow.dflow INFO: Standard output for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/9398fc64-df56-4fee-9da8-715f692c7171_isrForCrosstalkSources_1031728_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/9398fc64-df56-4fee-9da8-715f692c7171_isrForCrosstalkSources_1031728_25.stderr parsl.dataflow.dflow DEBUG: Task 1336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1336 has memoization hash eeebaf0447124b6c7461de7df0465874 parsl.dataflow.memoization INFO: Task 1336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1336 try 0 launched on executor multi with executor id 1337 parsl.dataflow.dflow INFO: Standard output for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5a78d395-2740-41e4-881d-03d9c5b521d3_isrForCrosstalkSources_1031701_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5a78d395-2740-41e4-881d-03d9c5b521d3_isrForCrosstalkSources_1031701_44.stderr parsl.dataflow.dflow DEBUG: Task 1337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1337 has memoization hash 6128c9c13b986b52611c1dca7fc31b0a parsl.dataflow.memoization INFO: Task 1337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1337 try 0 launched on executor multi with executor id 1338 parsl.dataflow.dflow INFO: Standard output for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/73bb2342-ecf6-48c2-88d6-a76861d3aadd_isrForCrosstalkSources_1031722_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/73bb2342-ecf6-48c2-88d6-a76861d3aadd_isrForCrosstalkSources_1031722_48.stderr parsl.dataflow.dflow DEBUG: Task 1338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1338 has memoization hash 11dd60eb894efe40c9b9a3afb5c87688 parsl.dataflow.memoization INFO: Task 1338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1338 try 0 launched on executor multi with executor id 1339 parsl.dataflow.dflow INFO: Standard output for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/67aff5ff-3fd9-4926-878b-9ac816a467c1_isrForCrosstalkSources_1031722_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/67aff5ff-3fd9-4926-878b-9ac816a467c1_isrForCrosstalkSources_1031722_30.stderr parsl.dataflow.dflow DEBUG: Task 1339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1339 has memoization hash 87100170e9b6e16bc92d8b49ed3d41f1 parsl.dataflow.memoization INFO: Task 1339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1339 try 0 launched on executor multi with executor id 1340 parsl.dataflow.dflow INFO: Standard output for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e610e7e5-a6bb-4c7a-8ef2-875900012d03_isrForCrosstalkSources_1031733_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e610e7e5-a6bb-4c7a-8ef2-875900012d03_isrForCrosstalkSources_1031733_39.stderr parsl.dataflow.dflow DEBUG: Task 1340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1340 has memoization hash 61c42b666746d75570a8d598ccec966a parsl.dataflow.memoization INFO: Task 1340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1340 try 0 launched on executor multi with executor id 1341 parsl.dataflow.dflow INFO: Standard output for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/6afcd16a-29dc-46d4-ba2d-faa9e4ab5a52_isrForCrosstalkSources_1031729_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/6afcd16a-29dc-46d4-ba2d-faa9e4ab5a52_isrForCrosstalkSources_1031729_58.stderr parsl.dataflow.dflow DEBUG: Task 1341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1341 has memoization hash e4554306e701768ecd6a1f1c50e7aadb parsl.dataflow.memoization INFO: Task 1341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1341 try 0 launched on executor multi with executor id 1342 parsl.dataflow.dflow INFO: Standard output for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/a583b80d-748e-4bea-b766-a22d8a919196_isrForCrosstalkSources_1031717_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/a583b80d-748e-4bea-b766-a22d8a919196_isrForCrosstalkSources_1031717_3.stderr parsl.dataflow.dflow DEBUG: Task 1342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1342 has memoization hash 508fa3f446ebdb2a32d5ddde356a9162 parsl.dataflow.memoization INFO: Task 1342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1342 try 0 launched on executor multi with executor id 1343 parsl.dataflow.dflow INFO: Standard output for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/44e05f87-d1f4-44e4-980a-c426723109e1_isrForCrosstalkSources_1031720_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/44e05f87-d1f4-44e4-980a-c426723109e1_isrForCrosstalkSources_1031720_19.stderr parsl.dataflow.dflow DEBUG: Task 1343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1343 has memoization hash 5ee9b40e7c68ca9a6bf136823c28b9f5 parsl.dataflow.memoization INFO: Task 1343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1343 try 0 launched on executor multi with executor id 1344 parsl.dataflow.dflow INFO: Standard output for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fb4fc5f6-fa96-4ce5-a485-a594599049ac_isrForCrosstalkSources_1031707_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fb4fc5f6-fa96-4ce5-a485-a594599049ac_isrForCrosstalkSources_1031707_54.stderr parsl.dataflow.dflow DEBUG: Task 1344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1344 has memoization hash c76bc8e585d8198fb6147f5ae594f15c parsl.dataflow.memoization INFO: Task 1344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1344 try 0 launched on executor multi with executor id 1345 parsl.dataflow.dflow INFO: Standard output for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ec9f0843-1034-49c7-8e18-7aaddc8b229e_isrForCrosstalkSources_1031724_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ec9f0843-1034-49c7-8e18-7aaddc8b229e_isrForCrosstalkSources_1031724_51.stderr parsl.dataflow.dflow DEBUG: Task 1345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1345 has memoization hash 113cf02f9465897f589bd70d574207cd parsl.dataflow.memoization INFO: Task 1345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1345 try 0 launched on executor multi with executor id 1346 parsl.dataflow.dflow INFO: Standard output for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7f95ab84-35ea-4ca1-af3c-4cfc54ff2bec_isrForCrosstalkSources_1031735_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7f95ab84-35ea-4ca1-af3c-4cfc54ff2bec_isrForCrosstalkSources_1031735_48.stderr parsl.dataflow.dflow DEBUG: Task 1346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1346 has memoization hash fe45ada98168950f474a74f2dada4346 parsl.dataflow.memoization INFO: Task 1346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1346 try 0 launched on executor multi with executor id 1347 parsl.dataflow.dflow INFO: Standard output for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c756d8df-ac05-4f1a-ad69-b984305d3812_isrForCrosstalkSources_1031715_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c756d8df-ac05-4f1a-ad69-b984305d3812_isrForCrosstalkSources_1031715_22.stderr parsl.dataflow.dflow DEBUG: Task 1347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1347 has memoization hash fa3c9cb06b2a0779a092478c43a91095 parsl.dataflow.memoization INFO: Task 1347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1347 try 0 launched on executor multi with executor id 1348 parsl.dataflow.dflow INFO: Standard output for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/8b04162c-9116-4665-803e-dc662060fdcd_isrForCrosstalkSources_1031731_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/8b04162c-9116-4665-803e-dc662060fdcd_isrForCrosstalkSources_1031731_23.stderr parsl.dataflow.dflow DEBUG: Task 1348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1348 has memoization hash 3d119017096fa7e8f2f39fc22a181113 parsl.dataflow.memoization INFO: Task 1348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1348 try 0 launched on executor multi with executor id 1349 parsl.dataflow.dflow INFO: Standard output for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/976a0810-31ac-4cac-be55-1501a7f7ab91_isrForCrosstalkSources_1031716_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/976a0810-31ac-4cac-be55-1501a7f7ab91_isrForCrosstalkSources_1031716_7.stderr parsl.dataflow.dflow DEBUG: Task 1349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1349 has memoization hash 20ed89d2aabb9d5cda5295aa83345992 parsl.dataflow.memoization INFO: Task 1349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1349 try 0 launched on executor multi with executor id 1350 parsl.dataflow.dflow INFO: Standard output for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ce047be-ea32-4d38-a378-7e02d64be75b_isrForCrosstalkSources_1031719_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ce047be-ea32-4d38-a378-7e02d64be75b_isrForCrosstalkSources_1031719_40.stderr parsl.dataflow.dflow DEBUG: Task 1350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1350 has memoization hash 290e0e20aadc83039f5d9e6e8919f1fc parsl.dataflow.memoization INFO: Task 1350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1350 try 0 launched on executor multi with executor id 1351 parsl.dataflow.dflow INFO: Standard output for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/00196f13-e10e-43a4-bf3d-e4c0edc113fe_isrForCrosstalkSources_1031709_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/00196f13-e10e-43a4-bf3d-e4c0edc113fe_isrForCrosstalkSources_1031709_9.stderr parsl.dataflow.dflow DEBUG: Task 1351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1351 has memoization hash 363746a740e061a63c45d1212193c8f0 parsl.dataflow.memoization INFO: Task 1351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1351 try 0 launched on executor multi with executor id 1352 parsl.dataflow.dflow INFO: Standard output for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/95b619e2-d8cc-4a1b-afb9-bbb17f36c02a_isrForCrosstalkSources_1031723_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/95b619e2-d8cc-4a1b-afb9-bbb17f36c02a_isrForCrosstalkSources_1031723_45.stderr parsl.dataflow.dflow DEBUG: Task 1352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1352 has memoization hash dfbca1b33c43314be955777f525d85db parsl.dataflow.memoization INFO: Task 1352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1352 try 0 launched on executor multi with executor id 1353 parsl.dataflow.dflow INFO: Standard output for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/2ccfcd09-f06e-4a60-a96f-feaf1b331e1c_isrForCrosstalkSources_1031716_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/2ccfcd09-f06e-4a60-a96f-feaf1b331e1c_isrForCrosstalkSources_1031716_6.stderr parsl.dataflow.dflow DEBUG: Task 1353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1353 has memoization hash d0b9d4e51b50b4a69d2b475e6549f5b8 parsl.dataflow.memoization INFO: Task 1353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1353 try 0 launched on executor multi with executor id 1354 parsl.dataflow.dflow INFO: Standard output for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/50127c9c-563c-40e9-932a-c73f444469ce_isrForCrosstalkSources_1031712_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/50127c9c-563c-40e9-932a-c73f444469ce_isrForCrosstalkSources_1031712_47.stderr parsl.dataflow.dflow DEBUG: Task 1354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1354 has memoization hash ce6a3a239698711f7849d5b5b46db687 parsl.dataflow.memoization INFO: Task 1354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1354 try 0 launched on executor multi with executor id 1355 parsl.dataflow.dflow INFO: Standard output for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/abad4e7e-b697-4fe1-8feb-dbcae4aa520a_isrForCrosstalkSources_1031720_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/abad4e7e-b697-4fe1-8feb-dbcae4aa520a_isrForCrosstalkSources_1031720_35.stderr parsl.dataflow.dflow DEBUG: Task 1355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1355 has memoization hash e230de0903a19996ee70edaa1e82237b parsl.dataflow.memoization INFO: Task 1355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1355 try 0 launched on executor multi with executor id 1356 parsl.dataflow.dflow INFO: Standard output for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a7f57bdb-85b9-4799-adf7-8437db88344f_isrForCrosstalkSources_1031728_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a7f57bdb-85b9-4799-adf7-8437db88344f_isrForCrosstalkSources_1031728_45.stderr parsl.dataflow.dflow DEBUG: Task 1356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1356 has memoization hash 741927ebfbd9c31ab79e6ed780212740 parsl.dataflow.memoization INFO: Task 1356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1356 try 0 launched on executor multi with executor id 1357 parsl.dataflow.dflow INFO: Standard output for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/c6656c0a-4e67-4361-8f18-178a88b9160d_isrForCrosstalkSources_1031717_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/c6656c0a-4e67-4361-8f18-178a88b9160d_isrForCrosstalkSources_1031717_17.stderr parsl.dataflow.dflow DEBUG: Task 1357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1357 has memoization hash 55a6ee47eb195e0cd67a247b3327d0e6 parsl.dataflow.memoization INFO: Task 1357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1357 try 0 launched on executor multi with executor id 1358 parsl.dataflow.dflow INFO: Standard output for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/963ae289-8e80-440e-a331-c349e79530b4_isrForCrosstalkSources_1031701_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/963ae289-8e80-440e-a331-c349e79530b4_isrForCrosstalkSources_1031701_62.stderr parsl.dataflow.dflow DEBUG: Task 1358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1358 has memoization hash 811c35e7e48d0c45eee1f723b069abab parsl.dataflow.memoization INFO: Task 1358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1358 try 0 launched on executor multi with executor id 1359 parsl.dataflow.dflow INFO: Standard output for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1b6c3c6a-1379-4da5-a553-abda853bc41c_isrForCrosstalkSources_1031711_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1b6c3c6a-1379-4da5-a553-abda853bc41c_isrForCrosstalkSources_1031711_50.stderr parsl.dataflow.dflow DEBUG: Task 1359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1359 has memoization hash e135730af17e83dddaf3ce2d58ad8453 parsl.dataflow.memoization INFO: Task 1359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1359 try 0 launched on executor multi with executor id 1360 parsl.dataflow.dflow INFO: Standard output for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/9554ebb9-0adb-4cde-98fb-3831892eaa3a_isrForCrosstalkSources_1031721_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/9554ebb9-0adb-4cde-98fb-3831892eaa3a_isrForCrosstalkSources_1031721_23.stderr parsl.dataflow.dflow DEBUG: Task 1360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1360 has memoization hash 185e8b9f0368595a5d2b857205c02d7f parsl.dataflow.memoization INFO: Task 1360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:13 proc_lsst.multi:170 [INFO] [multi] job_id 17147950 proc_lsst.multi INFO: [multi] job_id 17147950 2024-03-17 04:02:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-17 04:02:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147950 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147950 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17147950 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-17 04:02:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-17 04:02:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-17 04:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1360 try 0 launched on executor multi with executor id 1361 parsl.dataflow.dflow INFO: Standard output for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/2c5f1532-e2cb-4cd3-ae83-4f30d2c94ab2_isrForCrosstalkSources_1031714_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/2c5f1532-e2cb-4cd3-ae83-4f30d2c94ab2_isrForCrosstalkSources_1031714_32.stderr parsl.dataflow.dflow DEBUG: Task 1361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1361 has memoization hash b85ac5e6d5c14ff21c64c0e886ca00a3 parsl.dataflow.memoization INFO: Task 1361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1361 try 0 launched on executor multi with executor id 1362 parsl.dataflow.dflow INFO: Standard output for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/9d8eae95-8e56-4009-b154-f8f448f650ea_isrForCrosstalkSources_1031737_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/9d8eae95-8e56-4009-b154-f8f448f650ea_isrForCrosstalkSources_1031737_34.stderr parsl.dataflow.dflow DEBUG: Task 1362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1362 has memoization hash e98049930200bd84f342e34ceccfa798 parsl.dataflow.memoization INFO: Task 1362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fca80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1362 try 0 launched on executor multi with executor id 1363 parsl.dataflow.dflow INFO: Standard output for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/616d46cf-cbb7-47b8-a66b-e35155e71385_isrForCrosstalkSources_1031712_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/616d46cf-cbb7-47b8-a66b-e35155e71385_isrForCrosstalkSources_1031712_17.stderr parsl.dataflow.dflow DEBUG: Task 1363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1363 has memoization hash 917510ba5c5779ad9471b99c198c1581 parsl.dataflow.memoization INFO: Task 1363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1363 try 0 launched on executor multi with executor id 1364 parsl.dataflow.dflow INFO: Standard output for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/75b889bd-d94d-481f-9017-6524e6fb304b_isrForCrosstalkSources_1031728_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/75b889bd-d94d-481f-9017-6524e6fb304b_isrForCrosstalkSources_1031728_48.stderr parsl.dataflow.dflow DEBUG: Task 1364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1364 has memoization hash 6c5c598cd4c86303ca59a4e823df9410 parsl.dataflow.memoization INFO: Task 1364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1364 try 0 launched on executor multi with executor id 1365 parsl.dataflow.dflow INFO: Standard output for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/eee0f548-f462-4034-9ab7-623f7bd86f85_isrForCrosstalkSources_1031737_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/eee0f548-f462-4034-9ab7-623f7bd86f85_isrForCrosstalkSources_1031737_48.stderr parsl.dataflow.dflow DEBUG: Task 1365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1365 has memoization hash c0c08faa984e23615727aa6330fc5efc parsl.dataflow.memoization INFO: Task 1365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1365 try 0 launched on executor multi with executor id 1366 parsl.dataflow.dflow INFO: Standard output for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/936d5087-82e9-4fec-96fe-0d7de67311de_isrForCrosstalkSources_1031731_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/936d5087-82e9-4fec-96fe-0d7de67311de_isrForCrosstalkSources_1031731_38.stderr parsl.dataflow.dflow DEBUG: Task 1366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1366 has memoization hash 4e9f5b8aa88c96d7fb1a686bce28153b parsl.dataflow.memoization INFO: Task 1366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1366 try 0 launched on executor multi with executor id 1367 parsl.dataflow.dflow INFO: Standard output for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/38a17997-35f4-4e77-826b-8448d7961da2_isrForCrosstalkSources_1031711_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/38a17997-35f4-4e77-826b-8448d7961da2_isrForCrosstalkSources_1031711_10.stderr parsl.dataflow.dflow DEBUG: Task 1367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1367 has memoization hash 7259b1ecfe0525a641420bfe3c1fd515 parsl.dataflow.memoization INFO: Task 1367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1367 try 0 launched on executor multi with executor id 1368 parsl.dataflow.dflow INFO: Standard output for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d1ad88f7-bdcc-4c61-99ab-7a801039ae2c_isrForCrosstalkSources_1031730_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d1ad88f7-bdcc-4c61-99ab-7a801039ae2c_isrForCrosstalkSources_1031730_31.stderr parsl.dataflow.dflow DEBUG: Task 1368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1368 has memoization hash aa2f5163f3e519c05a6cdaedf8fd7711 parsl.dataflow.memoization INFO: Task 1368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1368 try 0 launched on executor multi with executor id 1369 parsl.dataflow.dflow INFO: Standard output for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d2072449-4001-4262-9e4b-2d147ac8fe23_isrForCrosstalkSources_1031725_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d2072449-4001-4262-9e4b-2d147ac8fe23_isrForCrosstalkSources_1031725_3.stderr parsl.dataflow.dflow DEBUG: Task 1369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1369 has memoization hash b7d24d6d3719efc991d689770c6348ef parsl.dataflow.memoization INFO: Task 1369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1369 try 0 launched on executor multi with executor id 1370 parsl.dataflow.dflow INFO: Standard output for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/5b468d03-3351-4027-919d-d2e2d2d55020_isrForCrosstalkSources_1031715_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/5b468d03-3351-4027-919d-d2e2d2d55020_isrForCrosstalkSources_1031715_47.stderr parsl.dataflow.dflow DEBUG: Task 1370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1370 has memoization hash 58a53d544bc10c6a2f4396665c3c4d05 parsl.dataflow.memoization INFO: Task 1370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1370 try 0 launched on executor multi with executor id 1371 parsl.dataflow.dflow INFO: Standard output for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/99aae279-adfe-44e8-b02c-c794b8feca0b_isrForCrosstalkSources_1031707_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/99aae279-adfe-44e8-b02c-c794b8feca0b_isrForCrosstalkSources_1031707_50.stderr parsl.dataflow.dflow DEBUG: Task 1371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1371 has memoization hash 3b8eefcf2050f06d8c6a9b7523bf2738 parsl.dataflow.memoization INFO: Task 1371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1371 try 0 launched on executor multi with executor id 1372 parsl.dataflow.dflow INFO: Standard output for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/4881429d-040c-4531-8f77-b2a15b9da7d9_isrForCrosstalkSources_1031724_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/4881429d-040c-4531-8f77-b2a15b9da7d9_isrForCrosstalkSources_1031724_13.stderr parsl.dataflow.dflow DEBUG: Task 1372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1372 has memoization hash 22faf0a33935d046d2237fac33db5c43 parsl.dataflow.memoization INFO: Task 1372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1372 try 0 launched on executor multi with executor id 1373 parsl.dataflow.dflow INFO: Standard output for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f5dd3e42-5e07-4ef5-9135-289c3783998f_isrForCrosstalkSources_1031706_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f5dd3e42-5e07-4ef5-9135-289c3783998f_isrForCrosstalkSources_1031706_57.stderr parsl.dataflow.dflow DEBUG: Task 1373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1373 has memoization hash 8496234f6d20bd0331b8a801c15c94f9 parsl.dataflow.memoization INFO: Task 1373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1373 try 0 launched on executor multi with executor id 1374 parsl.dataflow.dflow INFO: Standard output for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/19f9017d-55b2-4188-94bb-31fbbd95c52f_isrForCrosstalkSources_1031733_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/19f9017d-55b2-4188-94bb-31fbbd95c52f_isrForCrosstalkSources_1031733_5.stderr parsl.dataflow.dflow DEBUG: Task 1374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1374 has memoization hash a17a35ec92e15b369eccbcf528cead2a parsl.dataflow.memoization INFO: Task 1374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1374 try 0 launched on executor multi with executor id 1375 parsl.dataflow.dflow INFO: Standard output for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/da671006-1cf5-4250-8a39-3b446b81fec4_isrForCrosstalkSources_1031729_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/da671006-1cf5-4250-8a39-3b446b81fec4_isrForCrosstalkSources_1031729_20.stderr parsl.dataflow.dflow DEBUG: Task 1375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1375 has memoization hash 32451126a56f4f776fe5cf6be89dbf9b parsl.dataflow.memoization INFO: Task 1375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1375 try 0 launched on executor multi with executor id 1376 parsl.dataflow.dflow INFO: Standard output for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/d383a7d2-1006-4631-8a3b-dcaeea907fbe_isrForCrosstalkSources_1031717_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/d383a7d2-1006-4631-8a3b-dcaeea907fbe_isrForCrosstalkSources_1031717_19.stderr parsl.dataflow.dflow DEBUG: Task 1376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1376 has memoization hash b55ca4124e480cfe5868045a9123ad15 parsl.dataflow.memoization INFO: Task 1376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1376 try 0 launched on executor multi with executor id 1377 parsl.dataflow.dflow INFO: Standard output for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/12d38d0e-6a43-40bf-8d24-737b734beb83_isrForCrosstalkSources_1031731_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/12d38d0e-6a43-40bf-8d24-737b734beb83_isrForCrosstalkSources_1031731_16.stderr parsl.dataflow.dflow DEBUG: Task 1377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1377 has memoization hash a9e9c3d1645e97ff53b84cb4b600898c parsl.dataflow.memoization INFO: Task 1377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1377 try 0 launched on executor multi with executor id 1378 parsl.dataflow.dflow INFO: Standard output for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f2196306-2011-43f9-93cb-f310c9067180_isrForCrosstalkSources_1031703_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f2196306-2011-43f9-93cb-f310c9067180_isrForCrosstalkSources_1031703_32.stderr parsl.dataflow.dflow DEBUG: Task 1378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1378 has memoization hash 8dd5495541c8301fb7be6bfc6b244437 parsl.dataflow.memoization INFO: Task 1378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1378 try 0 launched on executor multi with executor id 1379 parsl.dataflow.dflow INFO: Standard output for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a21eafcb-195c-4ed9-ba06-1ac46b781a94_isrForCrosstalkSources_1031701_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a21eafcb-195c-4ed9-ba06-1ac46b781a94_isrForCrosstalkSources_1031701_47.stderr parsl.dataflow.dflow DEBUG: Task 1379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1379 has memoization hash 660b732b888d351e2437884076a7cdc4 parsl.dataflow.memoization INFO: Task 1379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1379 try 0 launched on executor multi with executor id 1380 parsl.dataflow.dflow INFO: Standard output for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/833218a8-8fcf-41fd-a7f7-ec4c52d6617f_isrForCrosstalkSources_1031728_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/833218a8-8fcf-41fd-a7f7-ec4c52d6617f_isrForCrosstalkSources_1031728_12.stderr parsl.dataflow.dflow DEBUG: Task 1380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1380 has memoization hash 969d0ccea9db7d4291bcba4a668f9e1c parsl.dataflow.memoization INFO: Task 1380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1380 try 0 launched on executor multi with executor id 1381 parsl.dataflow.dflow INFO: Standard output for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bc9fbc1f-b380-47a7-bcd1-2644d779d421_isrForCrosstalkSources_1031723_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bc9fbc1f-b380-47a7-bcd1-2644d779d421_isrForCrosstalkSources_1031723_9.stderr parsl.dataflow.dflow DEBUG: Task 1381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1381 has memoization hash 3ebab938cf763c80ea4f2d61905e3ec5 parsl.dataflow.memoization INFO: Task 1381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1381 try 0 launched on executor multi with executor id 1382 parsl.dataflow.dflow INFO: Standard output for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/ccd9db2b-a72a-49fd-9c49-f44278aa617b_isrForCrosstalkSources_1031735_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/ccd9db2b-a72a-49fd-9c49-f44278aa617b_isrForCrosstalkSources_1031735_57.stderr parsl.dataflow.dflow DEBUG: Task 1382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1382 has memoization hash c3ef7ec35434a9558637d923d7e0af98 parsl.dataflow.memoization INFO: Task 1382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa000e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1382 try 0 launched on executor multi with executor id 1383 parsl.dataflow.dflow INFO: Standard output for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/77924852-fc69-4f0e-bb07-12742aa97908_isrForCrosstalkSources_1031715_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/77924852-fc69-4f0e-bb07-12742aa97908_isrForCrosstalkSources_1031715_19.stderr parsl.dataflow.dflow DEBUG: Task 1383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1383 has memoization hash 6652218a062541f4bcb213ed2d803f26 parsl.dataflow.memoization INFO: Task 1383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1383 try 0 launched on executor multi with executor id 1384 parsl.dataflow.dflow INFO: Standard output for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/be4c5cdc-85bc-4f7c-b74d-88fc65a35588_isrForCrosstalkSources_1031702_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/be4c5cdc-85bc-4f7c-b74d-88fc65a35588_isrForCrosstalkSources_1031702_12.stderr parsl.dataflow.dflow DEBUG: Task 1384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1384 has memoization hash c54510b51031d60fec28ed706d1d32f3 parsl.dataflow.memoization INFO: Task 1384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1384 try 0 launched on executor multi with executor id 1385 parsl.dataflow.dflow INFO: Standard output for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/e5a4ecb9-3a8e-4684-ad38-7aa89a00dd45_isrForCrosstalkSources_1031705_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/e5a4ecb9-3a8e-4684-ad38-7aa89a00dd45_isrForCrosstalkSources_1031705_4.stderr parsl.dataflow.dflow DEBUG: Task 1385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1385 has memoization hash 205b80225594797a681b73a9a4bd33cc parsl.dataflow.memoization INFO: Task 1385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1385 try 0 launched on executor multi with executor id 1386 parsl.dataflow.dflow INFO: Standard output for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5f6ce825-0e86-4e6a-b702-dabc7e0fbe3a_isrForCrosstalkSources_1031711_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5f6ce825-0e86-4e6a-b702-dabc7e0fbe3a_isrForCrosstalkSources_1031711_17.stderr parsl.dataflow.dflow DEBUG: Task 1386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1386 has memoization hash 813e2d4ab555377d3e0a572cbeb31379 parsl.dataflow.memoization INFO: Task 1386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1386 try 0 launched on executor multi with executor id 1387 parsl.dataflow.dflow INFO: Standard output for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/32ec9f6a-aff7-4500-aefa-790624ea0996_isrForCrosstalkSources_1031705_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/32ec9f6a-aff7-4500-aefa-790624ea0996_isrForCrosstalkSources_1031705_20.stderr parsl.dataflow.dflow DEBUG: Task 1387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1387 has memoization hash f3efc4fc276062d3620476d5c786d779 parsl.dataflow.memoization INFO: Task 1387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1387 try 0 launched on executor multi with executor id 1388 parsl.dataflow.dflow INFO: Standard output for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/7c87490e-43b7-4aae-8e39-c95d85ed9c1f_isrForCrosstalkSources_1031737_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/7c87490e-43b7-4aae-8e39-c95d85ed9c1f_isrForCrosstalkSources_1031737_50.stderr parsl.dataflow.dflow DEBUG: Task 1388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1388 has memoization hash e208558ab0a29466388a3c3a0c90345a parsl.dataflow.memoization INFO: Task 1388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa002c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1388 try 0 launched on executor multi with executor id 1389 parsl.dataflow.dflow INFO: Standard output for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/def68b47-2985-4c51-be2d-37d829f98eee_isrForCrosstalkSources_1031706_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/def68b47-2985-4c51-be2d-37d829f98eee_isrForCrosstalkSources_1031706_21.stderr parsl.dataflow.dflow DEBUG: Task 1389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1389 has memoization hash 10735740433b2a821ac20f5d73a07d25 parsl.dataflow.memoization INFO: Task 1389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa004a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1389 try 0 launched on executor multi with executor id 1390 parsl.dataflow.dflow INFO: Standard output for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/06f47bbf-fe9a-40d9-9cf3-c3660e8dcaea_isrForCrosstalkSources_1031721_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/06f47bbf-fe9a-40d9-9cf3-c3660e8dcaea_isrForCrosstalkSources_1031721_42.stderr parsl.dataflow.dflow DEBUG: Task 1390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1390 has memoization hash 791306daf64d986eff09d5574b65bfdb parsl.dataflow.memoization INFO: Task 1390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1390 try 0 launched on executor multi with executor id 1391 parsl.dataflow.dflow INFO: Standard output for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/7e3d2aa4-ba2d-4fbb-9ca4-5307df8633c4_isrForCrosstalkSources_1031719_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/7e3d2aa4-ba2d-4fbb-9ca4-5307df8633c4_isrForCrosstalkSources_1031719_50.stderr parsl.dataflow.dflow DEBUG: Task 1391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1391 has memoization hash a900c9cbad50390e71bb2ea8fd362852 parsl.dataflow.memoization INFO: Task 1391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1391 try 0 launched on executor multi with executor id 1392 parsl.dataflow.dflow INFO: Standard output for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2890819b-b35e-4deb-a065-de70fab68261_isrForCrosstalkSources_1031704_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2890819b-b35e-4deb-a065-de70fab68261_isrForCrosstalkSources_1031704_20.stderr parsl.dataflow.dflow DEBUG: Task 1392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1392 has memoization hash ef073fe000eaa133871b1120649f3cce parsl.dataflow.memoization INFO: Task 1392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa005e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1392 try 0 launched on executor multi with executor id 1393 parsl.dataflow.dflow INFO: Standard output for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/41cb5dfa-4abe-4b19-8f72-e81f8542d8ee_isrForCrosstalkSources_1031727_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/41cb5dfa-4abe-4b19-8f72-e81f8542d8ee_isrForCrosstalkSources_1031727_46.stderr parsl.dataflow.dflow DEBUG: Task 1393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1393 has memoization hash 1278296b141dc4fcd677cdcd9d187f32 parsl.dataflow.memoization INFO: Task 1393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1393 try 0 launched on executor multi with executor id 1394 parsl.dataflow.dflow INFO: Standard output for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/7ec458af-0633-4006-89f0-02b0563b1178_isrForCrosstalkSources_1031722_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/7ec458af-0633-4006-89f0-02b0563b1178_isrForCrosstalkSources_1031722_55.stderr parsl.dataflow.dflow DEBUG: Task 1394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1394 has memoization hash 07beb7ab5cb0689f2c09d135530ab474 parsl.dataflow.memoization INFO: Task 1394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1394 try 0 launched on executor multi with executor id 1395 parsl.dataflow.dflow INFO: Standard output for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/518bcb97-a917-4058-87b7-d48c8f6b6ef5_isrForCrosstalkSources_1031699_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/518bcb97-a917-4058-87b7-d48c8f6b6ef5_isrForCrosstalkSources_1031699_45.stderr parsl.dataflow.dflow DEBUG: Task 1395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1395 has memoization hash f40ec3d20c4d63d04567f593753ed00c parsl.dataflow.memoization INFO: Task 1395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1395 try 0 launched on executor multi with executor id 1396 parsl.dataflow.dflow INFO: Standard output for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/dc88096e-cef6-49f4-b242-41f1f159a1f5_isrForCrosstalkSources_1031716_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/dc88096e-cef6-49f4-b242-41f1f159a1f5_isrForCrosstalkSources_1031716_34.stderr parsl.dataflow.dflow DEBUG: Task 1396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1396 has memoization hash 523ae5645fad7da5585732ed36eefd39 parsl.dataflow.memoization INFO: Task 1396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa007c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1396 try 0 launched on executor multi with executor id 1397 parsl.dataflow.dflow INFO: Standard output for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/7ced527e-84e0-44f9-b592-e115a6a629ed_isrForCrosstalkSources_1031699_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/7ced527e-84e0-44f9-b592-e115a6a629ed_isrForCrosstalkSources_1031699_33.stderr parsl.dataflow.dflow DEBUG: Task 1397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1397 has memoization hash d831b327813acfdcb176f2787b7c33db parsl.dataflow.memoization INFO: Task 1397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1397 try 0 launched on executor multi with executor id 1398 parsl.dataflow.dflow INFO: Standard output for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/77f6800a-06c0-42d2-af6e-c2c48ae79b28_isrForCrosstalkSources_1031713_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/77f6800a-06c0-42d2-af6e-c2c48ae79b28_isrForCrosstalkSources_1031713_55.stderr parsl.dataflow.dflow DEBUG: Task 1398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1398 has memoization hash b656489d7bb4f9750d0648a458806e0b parsl.dataflow.memoization INFO: Task 1398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa009a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1398 try 0 launched on executor multi with executor id 1399 parsl.dataflow.dflow INFO: Standard output for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c47d3aed-c5b0-49eb-9408-4e0de761a251_isrForCrosstalkSources_1031705_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c47d3aed-c5b0-49eb-9408-4e0de761a251_isrForCrosstalkSources_1031705_27.stderr parsl.dataflow.dflow DEBUG: Task 1399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1399 has memoization hash f1c22c70f95aa04708269f2f47c4c2ee parsl.dataflow.memoization INFO: Task 1399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1399 try 0 launched on executor multi with executor id 1400 parsl.dataflow.dflow INFO: Standard output for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/e8db40a5-3a92-424d-85e9-a7fd45a8bf39_isrForCrosstalkSources_1031735_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/e8db40a5-3a92-424d-85e9-a7fd45a8bf39_isrForCrosstalkSources_1031735_17.stderr parsl.dataflow.dflow DEBUG: Task 1400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1400 has memoization hash 976637cb8c92457e0dd76a75a8dd6749 parsl.dataflow.memoization INFO: Task 1400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1400 try 0 launched on executor multi with executor id 1401 parsl.dataflow.dflow INFO: Standard output for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3e79db8c-38fa-462c-b23e-68f74a24406b_isrForCrosstalkSources_1031736_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3e79db8c-38fa-462c-b23e-68f74a24406b_isrForCrosstalkSources_1031736_60.stderr parsl.dataflow.dflow DEBUG: Task 1401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1401 has memoization hash c477464b4a11d65c771766d1a3c33374 parsl.dataflow.memoization INFO: Task 1401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1401 try 0 launched on executor multi with executor id 1402 parsl.dataflow.dflow INFO: Standard output for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0aef29e4-4af7-4bf5-833a-55e0721bf234_isrForCrosstalkSources_1031721_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0aef29e4-4af7-4bf5-833a-55e0721bf234_isrForCrosstalkSources_1031721_16.stderr parsl.dataflow.dflow DEBUG: Task 1402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1402 has memoization hash 5030d14cb27476e142f93f0334c0a55b parsl.dataflow.memoization INFO: Task 1402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1402 try 0 launched on executor multi with executor id 1403 parsl.dataflow.dflow INFO: Standard output for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/69b3aa9f-1426-4168-b703-6e3db627a682_isrForCrosstalkSources_1031704_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/69b3aa9f-1426-4168-b703-6e3db627a682_isrForCrosstalkSources_1031704_32.stderr parsl.dataflow.dflow DEBUG: Task 1403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1403 has memoization hash 67b23e86885aa1427cee1d7609a03cfa parsl.dataflow.memoization INFO: Task 1403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1403 try 0 launched on executor multi with executor id 1404 parsl.dataflow.dflow INFO: Standard output for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/0a53845c-96bf-440f-87f6-1d9910605166_isrForCrosstalkSources_1031723_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/0a53845c-96bf-440f-87f6-1d9910605166_isrForCrosstalkSources_1031723_47.stderr parsl.dataflow.dflow DEBUG: Task 1404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1404 has memoization hash 081b344f0797a4e4021c70e0b3846af4 parsl.dataflow.memoization INFO: Task 1404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1404 try 0 launched on executor multi with executor id 1405 parsl.dataflow.dflow INFO: Standard output for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/b8ac035f-ec70-4671-b847-89d8eb561e2a_isrForCrosstalkSources_1031705_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/b8ac035f-ec70-4671-b847-89d8eb561e2a_isrForCrosstalkSources_1031705_28.stderr parsl.dataflow.dflow DEBUG: Task 1405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1405 has memoization hash c1f9e15dcd94734902a75d52caea10d5 parsl.dataflow.memoization INFO: Task 1405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1405 try 0 launched on executor multi with executor id 1406 parsl.dataflow.dflow INFO: Standard output for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/9da09d4a-a72d-4e8b-ab54-6df72bcb7d61_isrForCrosstalkSources_1031703_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/9da09d4a-a72d-4e8b-ab54-6df72bcb7d61_isrForCrosstalkSources_1031703_2.stderr parsl.dataflow.dflow DEBUG: Task 1406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1406 has memoization hash 81fed2b904d390ad0d396ce270e56bce parsl.dataflow.memoization INFO: Task 1406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1406 try 0 launched on executor multi with executor id 1407 parsl.dataflow.dflow INFO: Standard output for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/1c84bf66-64a5-4edd-96df-326388e7daf4_isrForCrosstalkSources_1031723_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/1c84bf66-64a5-4edd-96df-326388e7daf4_isrForCrosstalkSources_1031723_4.stderr parsl.dataflow.dflow DEBUG: Task 1407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1407 has memoization hash 0d8e8fff25da0932d6a384eb8de4bb72 parsl.dataflow.memoization INFO: Task 1407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1407 try 0 launched on executor multi with executor id 1408 parsl.dataflow.dflow INFO: Standard output for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4511f23c-8ad7-4670-9fbf-4755a6f3fe52_isrForCrosstalkSources_1031716_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4511f23c-8ad7-4670-9fbf-4755a6f3fe52_isrForCrosstalkSources_1031716_16.stderr parsl.dataflow.dflow DEBUG: Task 1408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1408 has memoization hash 1181496afccd59e46a527bca53dff823 parsl.dataflow.memoization INFO: Task 1408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1408 try 0 launched on executor multi with executor id 1409 parsl.dataflow.dflow INFO: Standard output for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ddcc9d8e-58a6-4e2b-ade6-87060a12e055_isrForCrosstalkSources_1031715_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ddcc9d8e-58a6-4e2b-ade6-87060a12e055_isrForCrosstalkSources_1031715_17.stderr parsl.dataflow.dflow DEBUG: Task 1409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1409 has memoization hash e1ff150b559ac779ba467c57f97eeb7a parsl.dataflow.memoization INFO: Task 1409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1409 try 0 launched on executor multi with executor id 1410 parsl.dataflow.dflow INFO: Standard output for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/14a5a416-df48-483e-bc12-115335f56fed_isrForCrosstalkSources_1031699_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/14a5a416-df48-483e-bc12-115335f56fed_isrForCrosstalkSources_1031699_2.stderr parsl.dataflow.dflow DEBUG: Task 1410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1410 has memoization hash 201ff3902fd4c36bfe67cc7eb1554d88 parsl.dataflow.memoization INFO: Task 1410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa00f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1410 try 0 launched on executor multi with executor id 1411 parsl.dataflow.dflow INFO: Standard output for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4f00fc12-09f4-437f-92a8-5aca19e0d5d8_isrForCrosstalkSources_1031703_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4f00fc12-09f4-437f-92a8-5aca19e0d5d8_isrForCrosstalkSources_1031703_33.stderr parsl.dataflow.dflow DEBUG: Task 1411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1411 has memoization hash 5c94f1e588a061cb619bee2d12cd7454 parsl.dataflow.memoization INFO: Task 1411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1411 try 0 launched on executor multi with executor id 1412 parsl.dataflow.dflow INFO: Standard output for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ff84eee7-828f-4b76-8b62-4717774915f7_isrForCrosstalkSources_1031708_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ff84eee7-828f-4b76-8b62-4717774915f7_isrForCrosstalkSources_1031708_6.stderr parsl.dataflow.dflow DEBUG: Task 1412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1412 has memoization hash 8862e96f17cd289891744d45b6f7cd6a parsl.dataflow.memoization INFO: Task 1412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa013a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1412 try 0 launched on executor multi with executor id 1413 parsl.dataflow.dflow INFO: Standard output for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/709f9cf4-5464-478e-b2cc-3e8250189553_isrForCrosstalkSources_1031725_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/709f9cf4-5464-478e-b2cc-3e8250189553_isrForCrosstalkSources_1031725_61.stderr parsl.dataflow.dflow DEBUG: Task 1413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1413 has memoization hash 580900e091415549cdb163caa0bef00c parsl.dataflow.memoization INFO: Task 1413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1413 try 0 launched on executor multi with executor id 1414 parsl.dataflow.dflow INFO: Standard output for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/6ad70a18-0e19-428b-bf71-9103ba309d74_isrForCrosstalkSources_1031712_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/6ad70a18-0e19-428b-bf71-9103ba309d74_isrForCrosstalkSources_1031712_43.stderr parsl.dataflow.dflow DEBUG: Task 1414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1414 has memoization hash 7b16d3cae3b968c5e20ab84e29e496f2 parsl.dataflow.memoization INFO: Task 1414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa011c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1414 try 0 launched on executor multi with executor id 1415 parsl.dataflow.dflow INFO: Standard output for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a2f9b1ac-20f6-4083-996d-d113b7db4da4_isrForCrosstalkSources_1031701_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a2f9b1ac-20f6-4083-996d-d113b7db4da4_isrForCrosstalkSources_1031701_31.stderr parsl.dataflow.dflow DEBUG: Task 1415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1415 has memoization hash a409cbc48e7599ae0decff030afa959e parsl.dataflow.memoization INFO: Task 1415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1415 try 0 launched on executor multi with executor id 1416 parsl.dataflow.dflow INFO: Standard output for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/b39c5816-b6d9-49f3-b2f2-2329f2e00226_isrForCrosstalkSources_1031725_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/b39c5816-b6d9-49f3-b2f2-2329f2e00226_isrForCrosstalkSources_1031725_4.stderr parsl.dataflow.dflow DEBUG: Task 1416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1416 has memoization hash a2965db6b3dbcd01289c0eab81b82b7f parsl.dataflow.memoization INFO: Task 1416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa014e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1416 try 0 launched on executor multi with executor id 1417 parsl.dataflow.dflow INFO: Standard output for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/26882130-1bd0-41b8-a8a8-e59a6c4bc422_isrForCrosstalkSources_1031722_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/26882130-1bd0-41b8-a8a8-e59a6c4bc422_isrForCrosstalkSources_1031722_58.stderr parsl.dataflow.dflow DEBUG: Task 1417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1417 has memoization hash f8577099cf9c12a215cb3de794c26372 parsl.dataflow.memoization INFO: Task 1417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1417 try 0 launched on executor multi with executor id 1418 parsl.dataflow.dflow INFO: Standard output for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/3da43e6d-2951-4852-b43b-bd6331d18f4d_isrForCrosstalkSources_1031714_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/3da43e6d-2951-4852-b43b-bd6331d18f4d_isrForCrosstalkSources_1031714_45.stderr parsl.dataflow.dflow DEBUG: Task 1418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1418 has memoization hash d9446ef4ce4c247d12314a0a6b9a8888 parsl.dataflow.memoization INFO: Task 1418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1418 try 0 launched on executor multi with executor id 1419 parsl.dataflow.dflow INFO: Standard output for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cd733fd3-9d76-4f7d-8c84-c9487da135cb_isrForCrosstalkSources_1031710_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cd733fd3-9d76-4f7d-8c84-c9487da135cb_isrForCrosstalkSources_1031710_3.stderr parsl.dataflow.dflow DEBUG: Task 1419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1419 has memoization hash 38605ca01c5011ee5aed4db0151398af parsl.dataflow.memoization INFO: Task 1419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa027a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1419 try 0 launched on executor multi with executor id 1420 parsl.dataflow.dflow INFO: Standard output for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c919a639-bd3d-406b-8e1a-2ce5d2bbee66_isrForCrosstalkSources_1031726_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c919a639-bd3d-406b-8e1a-2ce5d2bbee66_isrForCrosstalkSources_1031726_24.stderr parsl.dataflow.dflow DEBUG: Task 1420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1420 has memoization hash af9f1b501959c9855f214a86ade44a1a parsl.dataflow.memoization INFO: Task 1420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1420 try 0 launched on executor multi with executor id 1421 parsl.dataflow.dflow INFO: Standard output for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/49f70376-35e2-40a1-9117-41b47512acd5_isrForCrosstalkSources_1031701_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/49f70376-35e2-40a1-9117-41b47512acd5_isrForCrosstalkSources_1031701_22.stderr parsl.dataflow.dflow DEBUG: Task 1421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1421 has memoization hash 63f660501c82b6a658af818dd1f229f1 parsl.dataflow.memoization INFO: Task 1421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1421 try 0 launched on executor multi with executor id 1422 parsl.dataflow.dflow INFO: Standard output for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d5624d8a-154b-496a-9741-a2c37731a66b_isrForCrosstalkSources_1031726_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d5624d8a-154b-496a-9741-a2c37731a66b_isrForCrosstalkSources_1031726_19.stderr parsl.dataflow.dflow DEBUG: Task 1422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1422 has memoization hash 85d8e4b624a63b5ac0eafe096ce855fe parsl.dataflow.memoization INFO: Task 1422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1422 try 0 launched on executor multi with executor id 1423 parsl.dataflow.dflow INFO: Standard output for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d0a28a4a-2bad-49d1-9028-3341a8a1a33a_isrForCrosstalkSources_1031722_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d0a28a4a-2bad-49d1-9028-3341a8a1a33a_isrForCrosstalkSources_1031722_52.stderr parsl.dataflow.dflow DEBUG: Task 1423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1423 has memoization hash e8d1420de3f6217d1147a2316b6a60cd parsl.dataflow.memoization INFO: Task 1423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa025c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1423 try 0 launched on executor multi with executor id 1424 parsl.dataflow.dflow INFO: Standard output for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/78cfe645-cc96-420c-a8a2-c9de30dd0177_isrForCrosstalkSources_1031714_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/78cfe645-cc96-420c-a8a2-c9de30dd0177_isrForCrosstalkSources_1031714_56.stderr parsl.dataflow.dflow DEBUG: Task 1424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1424 has memoization hash b48a8b47e21b7bb36a4186b6f07b16bf parsl.dataflow.memoization INFO: Task 1424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1424 try 0 launched on executor multi with executor id 1425 parsl.dataflow.dflow INFO: Standard output for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/924e7665-feb2-4a09-87f6-be0dc978c18d_isrForCrosstalkSources_1031700_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/924e7665-feb2-4a09-87f6-be0dc978c18d_isrForCrosstalkSources_1031700_4.stderr parsl.dataflow.dflow DEBUG: Task 1425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1425 has memoization hash 9191fd641f2dea1eabe48f9b43e111c9 parsl.dataflow.memoization INFO: Task 1425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa023e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1425 try 0 launched on executor multi with executor id 1426 parsl.dataflow.dflow INFO: Standard output for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bab9a51e-c698-4cf4-8e6d-324a4650aa39_isrForCrosstalkSources_1031734_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bab9a51e-c698-4cf4-8e6d-324a4650aa39_isrForCrosstalkSources_1031734_47.stderr parsl.dataflow.dflow DEBUG: Task 1426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1426 has memoization hash 5624eddcb6176f53475095ad1f158a7d parsl.dataflow.memoization INFO: Task 1426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1426 try 0 launched on executor multi with executor id 1427 parsl.dataflow.dflow INFO: Standard output for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e03ca61f-2822-4986-83b4-0e793339e5a1_isrForCrosstalkSources_1031730_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e03ca61f-2822-4986-83b4-0e793339e5a1_isrForCrosstalkSources_1031730_52.stderr parsl.dataflow.dflow DEBUG: Task 1427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1427 has memoization hash a988e2f845889a07f3a629dc55daaa5a parsl.dataflow.memoization INFO: Task 1427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1427 try 0 launched on executor multi with executor id 1428 parsl.dataflow.dflow INFO: Standard output for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/981fd569-5b5c-4d72-a8b1-750657d4a653_isrForCrosstalkSources_1031736_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/981fd569-5b5c-4d72-a8b1-750657d4a653_isrForCrosstalkSources_1031736_53.stderr parsl.dataflow.dflow DEBUG: Task 1428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1428 has memoization hash c3410c309b43660e96aad0fcd0ca8d8e parsl.dataflow.memoization INFO: Task 1428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa022a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1428 try 0 launched on executor multi with executor id 1429 parsl.dataflow.dflow INFO: Standard output for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a5a8154f-b128-4bbe-8d9e-846022eaf3f3_isrForCrosstalkSources_1031709_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a5a8154f-b128-4bbe-8d9e-846022eaf3f3_isrForCrosstalkSources_1031709_6.stderr parsl.dataflow.dflow DEBUG: Task 1429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1429 has memoization hash 8782376623e2d5ffd283b39b223bcb82 parsl.dataflow.memoization INFO: Task 1429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1429 try 0 launched on executor multi with executor id 1430 parsl.dataflow.dflow INFO: Standard output for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/29bab00b-7c58-4d0d-b2e6-606267d6099c_isrForCrosstalkSources_1031728_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/29bab00b-7c58-4d0d-b2e6-606267d6099c_isrForCrosstalkSources_1031728_15.stderr parsl.dataflow.dflow DEBUG: Task 1430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1430 has memoization hash 3a08a2c82a24605e1ce714d1934ede1c parsl.dataflow.memoization INFO: Task 1430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa020c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1430 try 0 launched on executor multi with executor id 1431 parsl.dataflow.dflow INFO: Standard output for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8147d972-eaaf-48ac-b052-f383dec9ac92_isrForCrosstalkSources_1031714_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8147d972-eaaf-48ac-b052-f383dec9ac92_isrForCrosstalkSources_1031714_6.stderr parsl.dataflow.dflow DEBUG: Task 1431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1431 has memoization hash f566d82edb95d8dff2f0856857527595 parsl.dataflow.memoization INFO: Task 1431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1431 try 0 launched on executor multi with executor id 1432 parsl.dataflow.dflow INFO: Standard output for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/19e22ecd-c7ae-475d-950b-f997aa5dd725_isrForCrosstalkSources_1031722_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/19e22ecd-c7ae-475d-950b-f997aa5dd725_isrForCrosstalkSources_1031722_4.stderr parsl.dataflow.dflow DEBUG: Task 1432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1432 has memoization hash 3c2178945580a200f148cdd639fbda5a parsl.dataflow.memoization INFO: Task 1432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1432 try 0 launched on executor multi with executor id 1433 parsl.dataflow.dflow INFO: Standard output for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4391da84-2813-4a68-9e2c-03439f0bfed2_isrForCrosstalkSources_1031715_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4391da84-2813-4a68-9e2c-03439f0bfed2_isrForCrosstalkSources_1031715_31.stderr parsl.dataflow.dflow DEBUG: Task 1433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1433 has memoization hash f6488faf8bdcb9816220f4fbbd45c6a6 parsl.dataflow.memoization INFO: Task 1433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1433 try 0 launched on executor multi with executor id 1434 parsl.dataflow.dflow INFO: Standard output for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/54d1b2f6-7cd4-4d39-91f9-5ee17b9e4849_isrForCrosstalkSources_1031718_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/54d1b2f6-7cd4-4d39-91f9-5ee17b9e4849_isrForCrosstalkSources_1031718_61.stderr parsl.dataflow.dflow DEBUG: Task 1434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1434 has memoization hash 4bb4cb17e4de7aa975b9058b2f5b97f1 parsl.dataflow.memoization INFO: Task 1434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1434 try 0 launched on executor multi with executor id 1435 parsl.dataflow.dflow INFO: Standard output for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/809d18d7-ccf1-43e5-9520-e263abde32de_isrForCrosstalkSources_1031731_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/809d18d7-ccf1-43e5-9520-e263abde32de_isrForCrosstalkSources_1031731_11.stderr parsl.dataflow.dflow DEBUG: Task 1435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1435 has memoization hash 95aa581bbed0876b3089970417a76583 parsl.dataflow.memoization INFO: Task 1435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1435 try 0 launched on executor multi with executor id 1436 parsl.dataflow.dflow INFO: Standard output for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4daffc18-34c6-403a-ac91-69dbe1de904e_isrForCrosstalkSources_1031699_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4daffc18-34c6-403a-ac91-69dbe1de904e_isrForCrosstalkSources_1031699_20.stderr parsl.dataflow.dflow DEBUG: Task 1436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1436 has memoization hash 5bf3c1eb26d0d2c968a518633f5bdcf4 parsl.dataflow.memoization INFO: Task 1436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1436 try 0 launched on executor multi with executor id 1437 parsl.dataflow.dflow INFO: Standard output for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/fa338d43-c26b-489a-8dcd-390d09674d24_isrForCrosstalkSources_1031734_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/fa338d43-c26b-489a-8dcd-390d09674d24_isrForCrosstalkSources_1031734_46.stderr parsl.dataflow.dflow DEBUG: Task 1437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1437 has memoization hash 08156a5244dda42801ff3b2995dad075 parsl.dataflow.memoization INFO: Task 1437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1437 try 0 launched on executor multi with executor id 1438 parsl.dataflow.dflow INFO: Standard output for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/284d5be8-4144-4b84-bf59-fdc6ed3f1607_isrForCrosstalkSources_1031708_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/284d5be8-4144-4b84-bf59-fdc6ed3f1607_isrForCrosstalkSources_1031708_50.stderr parsl.dataflow.dflow DEBUG: Task 1438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1438 has memoization hash 782bc49fa885724488d938e62eb98fd5 parsl.dataflow.memoization INFO: Task 1438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1438 try 0 launched on executor multi with executor id 1439 parsl.dataflow.dflow INFO: Standard output for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/009458f5-ea18-465a-bf1c-4c1ad5a75f65_isrForCrosstalkSources_1031706_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/009458f5-ea18-465a-bf1c-4c1ad5a75f65_isrForCrosstalkSources_1031706_61.stderr parsl.dataflow.dflow DEBUG: Task 1439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1439 has memoization hash 005eb81929188f1754f7cd54310ec40f parsl.dataflow.memoization INFO: Task 1439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1439 try 0 launched on executor multi with executor id 1440 parsl.dataflow.dflow INFO: Standard output for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/42f2f266-aa06-4f4d-b397-0e289c8e57bf_isrForCrosstalkSources_1031720_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/42f2f266-aa06-4f4d-b397-0e289c8e57bf_isrForCrosstalkSources_1031720_52.stderr parsl.dataflow.dflow DEBUG: Task 1440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1440 has memoization hash f01cc39ef5d95a83b4ce1ee1959fa36e parsl.dataflow.memoization INFO: Task 1440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1440 try 0 launched on executor multi with executor id 1441 parsl.dataflow.dflow INFO: Standard output for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/cdde7a2c-5464-49b7-a71e-955d728ba451_isrForCrosstalkSources_1031705_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/cdde7a2c-5464-49b7-a71e-955d728ba451_isrForCrosstalkSources_1031705_17.stderr parsl.dataflow.dflow DEBUG: Task 1441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1441 has memoization hash 5969bd27c93f9d160e84459538790ce7 parsl.dataflow.memoization INFO: Task 1441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa019e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1441 try 0 launched on executor multi with executor id 1442 parsl.dataflow.dflow INFO: Standard output for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/b50949a7-bdd4-49a3-9155-5c80ae45a507_isrForCrosstalkSources_1031727_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/b50949a7-bdd4-49a3-9155-5c80ae45a507_isrForCrosstalkSources_1031727_42.stderr parsl.dataflow.dflow DEBUG: Task 1442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1442 has memoization hash 872de2820d019bc8c0beebd6ce315d76 parsl.dataflow.memoization INFO: Task 1442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa018a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1442 try 0 launched on executor multi with executor id 1443 parsl.dataflow.dflow INFO: Standard output for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2ba3cc97-ac4b-4af4-a742-2488ace65aa0_isrForCrosstalkSources_1031734_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2ba3cc97-ac4b-4af4-a742-2488ace65aa0_isrForCrosstalkSources_1031734_16.stderr parsl.dataflow.dflow DEBUG: Task 1443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1443 has memoization hash 069a99d20184cb8b9fe44ed65de92552 parsl.dataflow.memoization INFO: Task 1443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1443 try 0 launched on executor multi with executor id 1444 parsl.dataflow.dflow INFO: Standard output for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/11e1b0b7-9ca5-4998-891d-6ba28d64858b_isrForCrosstalkSources_1031724_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/11e1b0b7-9ca5-4998-891d-6ba28d64858b_isrForCrosstalkSources_1031724_17.stderr parsl.dataflow.dflow DEBUG: Task 1444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1444 has memoization hash 910b79ae3c642b7e6972b100eccbf35e parsl.dataflow.memoization INFO: Task 1444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1444 try 0 launched on executor multi with executor id 1445 parsl.dataflow.dflow INFO: Standard output for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c2a3002a-b090-44ad-9136-a61a6ff5ccb8_isrForCrosstalkSources_1031706_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c2a3002a-b090-44ad-9136-a61a6ff5ccb8_isrForCrosstalkSources_1031706_37.stderr parsl.dataflow.dflow DEBUG: Task 1445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1445 has memoization hash 16fa03ed2e4f12c69d488eab0ed23dd9 parsl.dataflow.memoization INFO: Task 1445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1445 try 0 launched on executor multi with executor id 1446 parsl.dataflow.dflow INFO: Standard output for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/9513a4f5-6c05-4c25-b7a4-afe798fd3108_isrForCrosstalkSources_1031709_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/9513a4f5-6c05-4c25-b7a4-afe798fd3108_isrForCrosstalkSources_1031709_16.stderr parsl.dataflow.dflow DEBUG: Task 1446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1446 has memoization hash 0ee1c7eac0842e7fabe9d50285dcfee1 parsl.dataflow.memoization INFO: Task 1446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa016c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1446 try 0 launched on executor multi with executor id 1447 parsl.dataflow.dflow INFO: Standard output for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/57b5bca3-014c-4cc7-a453-061a65f46c09_isrForCrosstalkSources_1031717_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/57b5bca3-014c-4cc7-a453-061a65f46c09_isrForCrosstalkSources_1031717_28.stderr parsl.dataflow.dflow DEBUG: Task 1447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1447 has memoization hash ffd0234fe173fa72303a53988ebf6747 parsl.dataflow.memoization INFO: Task 1447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1447 try 0 launched on executor multi with executor id 1448 parsl.dataflow.dflow INFO: Standard output for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/97cabd04-f316-4a3f-91ae-02167709003a_isrForCrosstalkSources_1031721_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/97cabd04-f316-4a3f-91ae-02167709003a_isrForCrosstalkSources_1031721_43.stderr parsl.dataflow.dflow DEBUG: Task 1448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1448 has memoization hash 92317d6a9ec6b2c776a23cc26fa825d4 parsl.dataflow.memoization INFO: Task 1448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa01580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1448 try 0 launched on executor multi with executor id 1449 parsl.dataflow.dflow INFO: Standard output for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/c1decf57-e3da-49a5-9444-bc5c67c4a505_isrForCrosstalkSources_1031704_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/c1decf57-e3da-49a5-9444-bc5c67c4a505_isrForCrosstalkSources_1031704_29.stderr parsl.dataflow.dflow DEBUG: Task 1449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1449 has memoization hash 95739c3d7d5ce0ae41d002f94313cd3e parsl.dataflow.memoization INFO: Task 1449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1449 try 0 launched on executor multi with executor id 1450 parsl.dataflow.dflow INFO: Standard output for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/92b57920-0a32-4131-b47a-09ddee6aa8b1_isrForCrosstalkSources_1031716_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/92b57920-0a32-4131-b47a-09ddee6aa8b1_isrForCrosstalkSources_1031716_17.stderr parsl.dataflow.dflow DEBUG: Task 1450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1450 has memoization hash dda462a4a61fbf21b3d06d8f1bb4d810 parsl.dataflow.memoization INFO: Task 1450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1450 try 0 launched on executor multi with executor id 1451 parsl.dataflow.dflow INFO: Standard output for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7f9f1b5b-d03c-4231-ab97-9243913ce0bf_isrForCrosstalkSources_1031729_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7f9f1b5b-d03c-4231-ab97-9243913ce0bf_isrForCrosstalkSources_1031729_3.stderr parsl.dataflow.dflow DEBUG: Task 1451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1451 has memoization hash c061adbf72c0336d261083751369f75b parsl.dataflow.memoization INFO: Task 1451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1451 try 0 launched on executor multi with executor id 1452 parsl.dataflow.dflow INFO: Standard output for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/68542543-df2f-4c53-b9ee-b5ba5d55cfbf_isrForCrosstalkSources_1031703_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/68542543-df2f-4c53-b9ee-b5ba5d55cfbf_isrForCrosstalkSources_1031703_22.stderr parsl.dataflow.dflow DEBUG: Task 1452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1452 has memoization hash cf6207aa1738767b598e27099b785b58 parsl.dataflow.memoization INFO: Task 1452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1452 try 0 launched on executor multi with executor id 1453 parsl.dataflow.dflow INFO: Standard output for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/b04a5a10-332b-4785-8e23-cfb10f18d61f_isrForCrosstalkSources_1031701_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/b04a5a10-332b-4785-8e23-cfb10f18d61f_isrForCrosstalkSources_1031701_33.stderr parsl.dataflow.dflow DEBUG: Task 1453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1453 has memoization hash 8154a2f63345447fa869e9c68790299e parsl.dataflow.memoization INFO: Task 1453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1453 try 0 launched on executor multi with executor id 1454 parsl.dataflow.dflow INFO: Standard output for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cef56e85-10aa-44e4-9b99-365063e56b29_isrForCrosstalkSources_1031710_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cef56e85-10aa-44e4-9b99-365063e56b29_isrForCrosstalkSources_1031710_40.stderr parsl.dataflow.dflow DEBUG: Task 1454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1454 has memoization hash fc57689faf0ac703ba8bc5cb98327e09 parsl.dataflow.memoization INFO: Task 1454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa02fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1454 try 0 launched on executor multi with executor id 1455 parsl.dataflow.dflow INFO: Standard output for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/2ca39469-e244-4618-9d8d-9d9d7fc0c1b9_isrForCrosstalkSources_1031721_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/2ca39469-e244-4618-9d8d-9d9d7fc0c1b9_isrForCrosstalkSources_1031721_22.stderr parsl.dataflow.dflow DEBUG: Task 1455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1455 has memoization hash 2c9c86eb0fa908463c00f1c830c6c87f parsl.dataflow.memoization INFO: Task 1455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1455 try 0 launched on executor multi with executor id 1456 parsl.dataflow.dflow INFO: Standard output for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/43e87c8a-6ddc-49c0-b38f-eafd387be476_isrForCrosstalkSources_1031734_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/43e87c8a-6ddc-49c0-b38f-eafd387be476_isrForCrosstalkSources_1031734_26.stderr parsl.dataflow.dflow DEBUG: Task 1456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1456 has memoization hash 436828e64d05470389b13d2380912ec8 parsl.dataflow.memoization INFO: Task 1456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1456 try 0 launched on executor multi with executor id 1457 parsl.dataflow.dflow INFO: Standard output for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/8ce63b30-89ee-4391-bfd9-76dccd77b5dc_isrForCrosstalkSources_1031711_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/8ce63b30-89ee-4391-bfd9-76dccd77b5dc_isrForCrosstalkSources_1031711_36.stderr parsl.dataflow.dflow DEBUG: Task 1457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1457 has memoization hash 097eb7da21ec2fc281604ac3bc16d1a8 parsl.dataflow.memoization INFO: Task 1457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1457 try 0 launched on executor multi with executor id 1458 parsl.dataflow.dflow INFO: Standard output for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c2318c6f-c86f-4b79-bdc0-787ab0b6d68f_isrForCrosstalkSources_1031722_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c2318c6f-c86f-4b79-bdc0-787ab0b6d68f_isrForCrosstalkSources_1031722_2.stderr parsl.dataflow.dflow DEBUG: Task 1458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1458 has memoization hash 9d3a9056e213e7b67419f1b930973f77 parsl.dataflow.memoization INFO: Task 1458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa031a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1458 try 0 launched on executor multi with executor id 1459 parsl.dataflow.dflow INFO: Standard output for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/e86fb97b-348f-448f-adc2-abbb16164dd8_isrForCrosstalkSources_1031732_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/e86fb97b-348f-448f-adc2-abbb16164dd8_isrForCrosstalkSources_1031732_3.stderr parsl.dataflow.dflow DEBUG: Task 1459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1459 has memoization hash 9f744fa553b49aa1d36261e5d5235247 parsl.dataflow.memoization INFO: Task 1459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa032e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1459 try 0 launched on executor multi with executor id 1460 parsl.dataflow.dflow INFO: Standard output for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d80f9f63-692a-42de-9950-ece2bbfdfb66_isrForCrosstalkSources_1031737_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d80f9f63-692a-42de-9950-ece2bbfdfb66_isrForCrosstalkSources_1031737_24.stderr parsl.dataflow.dflow DEBUG: Task 1460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1460 has memoization hash cf003393001f74cc35a305d5d08ecd2a parsl.dataflow.memoization INFO: Task 1460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1460 try 0 launched on executor multi with executor id 1461 parsl.dataflow.dflow INFO: Standard output for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/77feddb7-0442-4d96-88a1-7f04d59ac5bf_isrForCrosstalkSources_1031725_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/77feddb7-0442-4d96-88a1-7f04d59ac5bf_isrForCrosstalkSources_1031725_62.stderr parsl.dataflow.dflow DEBUG: Task 1461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1461 has memoization hash 0ae0369565a187e3cf32b0ba35a5a524 parsl.dataflow.memoization INFO: Task 1461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1461 try 0 launched on executor multi with executor id 1462 parsl.dataflow.dflow INFO: Standard output for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5918cfc7-4dd2-40be-b5db-9660f0ff2c88_isrForCrosstalkSources_1031735_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5918cfc7-4dd2-40be-b5db-9660f0ff2c88_isrForCrosstalkSources_1031735_43.stderr parsl.dataflow.dflow DEBUG: Task 1462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1462 has memoization hash 5d6390743375cadfc9d0bdd3bc2a5cfc parsl.dataflow.memoization INFO: Task 1462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1462 try 0 launched on executor multi with executor id 1463 parsl.dataflow.dflow INFO: Standard output for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/abc33104-5273-46a9-813e-7989f8905d1e_isrForCrosstalkSources_1031730_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/abc33104-5273-46a9-813e-7989f8905d1e_isrForCrosstalkSources_1031730_28.stderr parsl.dataflow.dflow DEBUG: Task 1463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1463 has memoization hash ec79adaa6896bb416f33fcd6f76793f6 parsl.dataflow.memoization INFO: Task 1463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa034c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1463 try 0 launched on executor multi with executor id 1464 parsl.dataflow.dflow INFO: Standard output for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e854402f-4b39-4981-bac2-559891dd864f_isrForCrosstalkSources_1031704_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e854402f-4b39-4981-bac2-559891dd864f_isrForCrosstalkSources_1031704_17.stderr parsl.dataflow.dflow DEBUG: Task 1464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1464 has memoization hash fab2d1ad33745f9c12eb7dd0b5cb2da8 parsl.dataflow.memoization INFO: Task 1464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1464 try 0 launched on executor multi with executor id 1465 parsl.dataflow.dflow INFO: Standard output for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/10c7bf26-7c2b-4fa7-9b18-9aba35e43558_isrForCrosstalkSources_1031700_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/10c7bf26-7c2b-4fa7-9b18-9aba35e43558_isrForCrosstalkSources_1031700_55.stderr parsl.dataflow.dflow DEBUG: Task 1465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1465 has memoization hash 64afd691656bcc781d5d6635f0db9262 parsl.dataflow.memoization INFO: Task 1465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa036a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1465 try 0 launched on executor multi with executor id 1466 parsl.dataflow.dflow INFO: Standard output for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/4bd019b6-c100-45de-bff9-eddadf85a392_isrForCrosstalkSources_1031708_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/4bd019b6-c100-45de-bff9-eddadf85a392_isrForCrosstalkSources_1031708_62.stderr parsl.dataflow.dflow DEBUG: Task 1466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1466 has memoization hash 8d3768e4487ad91d3ab715b8cf99ccc8 parsl.dataflow.memoization INFO: Task 1466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1466 try 0 launched on executor multi with executor id 1467 parsl.dataflow.dflow INFO: Standard output for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a9485acb-61fe-4b6a-9b8c-53eae4d4571d_isrForCrosstalkSources_1031730_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a9485acb-61fe-4b6a-9b8c-53eae4d4571d_isrForCrosstalkSources_1031730_18.stderr parsl.dataflow.dflow DEBUG: Task 1467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1467 has memoization hash 800184e79683e9c2d2e19798415aad44 parsl.dataflow.memoization INFO: Task 1467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1467 try 0 launched on executor multi with executor id 1468 parsl.dataflow.dflow INFO: Standard output for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8c650af3-2bdc-4640-bafa-f7f8b3dce07a_isrForCrosstalkSources_1031714_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8c650af3-2bdc-4640-bafa-f7f8b3dce07a_isrForCrosstalkSources_1031714_15.stderr parsl.dataflow.dflow DEBUG: Task 1468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1468 has memoization hash 5d48e6bc41f11b14e2d4711629f1dc59 parsl.dataflow.memoization INFO: Task 1468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa037e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1468 try 0 launched on executor multi with executor id 1469 parsl.dataflow.dflow INFO: Standard output for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/309fad20-7589-4033-b40f-07d9c31f5103_isrForCrosstalkSources_1031710_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/309fad20-7589-4033-b40f-07d9c31f5103_isrForCrosstalkSources_1031710_56.stderr parsl.dataflow.dflow DEBUG: Task 1469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1469 has memoization hash e08b0e2eed9258824d71fa63f305c21b parsl.dataflow.memoization INFO: Task 1469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:15 proc_lsst.multi:170 [INFO] [multi] job_id 17147951 proc_lsst.multi INFO: [multi] job_id 17147951 2024-03-17 04:02:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:02:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147951 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147951 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17147951 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-17 04:02:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-17 04:02:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1469 try 0 launched on executor multi with executor id 1470 parsl.dataflow.dflow INFO: Standard output for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/1885307d-8fcc-454c-8b8a-d9a40ba04849_isrForCrosstalkSources_1031725_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/1885307d-8fcc-454c-8b8a-d9a40ba04849_isrForCrosstalkSources_1031725_58.stderr parsl.dataflow.dflow DEBUG: Task 1470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1470 has memoization hash 444ae032a296c089d8ee576c54920f5d parsl.dataflow.memoization INFO: Task 1470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1470 try 0 launched on executor multi with executor id 1471 parsl.dataflow.dflow INFO: Standard output for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/1c40de5d-6e3e-4686-ad59-f95b9653de83_isrForCrosstalkSources_1031729_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/1c40de5d-6e3e-4686-ad59-f95b9653de83_isrForCrosstalkSources_1031729_57.stderr parsl.dataflow.dflow DEBUG: Task 1471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1471 has memoization hash 0f6d8c26eab4e3067a5774aa9cf94ca3 parsl.dataflow.memoization INFO: Task 1471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1471 try 0 launched on executor multi with executor id 1472 parsl.dataflow.dflow INFO: Standard output for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/af5c3141-4522-4783-bbc8-f70104e05676_isrForCrosstalkSources_1031711_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/af5c3141-4522-4783-bbc8-f70104e05676_isrForCrosstalkSources_1031711_22.stderr parsl.dataflow.dflow DEBUG: Task 1472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1472 has memoization hash 9cf56113e2227ff23b18e6e26f019f8c parsl.dataflow.memoization INFO: Task 1472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1472 try 0 launched on executor multi with executor id 1473 parsl.dataflow.dflow INFO: Standard output for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/26672e3c-57aa-4819-b1f1-70ebf48bf1ea_isrForCrosstalkSources_1031728_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/26672e3c-57aa-4819-b1f1-70ebf48bf1ea_isrForCrosstalkSources_1031728_19.stderr parsl.dataflow.dflow DEBUG: Task 1473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1473 has memoization hash 9b8208f5596804579d0a917feac3bd18 parsl.dataflow.memoization INFO: Task 1473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fb5c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1473 try 0 launched on executor multi with executor id 1474 parsl.dataflow.dflow INFO: Standard output for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0bdba1ed-444c-4a39-b720-b033701e80f8_isrForCrosstalkSources_1031732_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0bdba1ed-444c-4a39-b720-b033701e80f8_isrForCrosstalkSources_1031732_42.stderr parsl.dataflow.dflow DEBUG: Task 1474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1474 has memoization hash a3a6a3e0fb0275cc4602217152f7ca8e parsl.dataflow.memoization INFO: Task 1474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1474 try 0 launched on executor multi with executor id 1475 parsl.dataflow.dflow INFO: Standard output for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/11b410f1-9df9-4a6a-a4ee-9b605168bf73_isrForCrosstalkSources_1031724_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/11b410f1-9df9-4a6a-a4ee-9b605168bf73_isrForCrosstalkSources_1031724_18.stderr parsl.dataflow.dflow DEBUG: Task 1475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1475 has memoization hash e3e0a296a15996ba05a650aa3a92f619 parsl.dataflow.memoization INFO: Task 1475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1475 try 0 launched on executor multi with executor id 1476 parsl.dataflow.dflow INFO: Standard output for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fcc1cf16-d754-44ad-9c81-43cefb170d20_isrForCrosstalkSources_1031729_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fcc1cf16-d754-44ad-9c81-43cefb170d20_isrForCrosstalkSources_1031729_32.stderr parsl.dataflow.dflow DEBUG: Task 1476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1476 has memoization hash 3e7e876a730d49ee3c4bfd4e1e88caa3 parsl.dataflow.memoization INFO: Task 1476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1476 try 0 launched on executor multi with executor id 1477 parsl.dataflow.dflow INFO: Standard output for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/f3ea38f0-d265-4644-8f06-aa740b49806e_isrForCrosstalkSources_1031704_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/f3ea38f0-d265-4644-8f06-aa740b49806e_isrForCrosstalkSources_1031704_58.stderr parsl.dataflow.dflow DEBUG: Task 1477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1477 has memoization hash 4aa4ce42541db11f66ab2cfea7ed5e24 parsl.dataflow.memoization INFO: Task 1477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1477 try 0 launched on executor multi with executor id 1478 parsl.dataflow.dflow INFO: Standard output for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/2a025670-c3d5-4382-9a21-cca0e61776b4_isrForCrosstalkSources_1031713_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/2a025670-c3d5-4382-9a21-cca0e61776b4_isrForCrosstalkSources_1031713_60.stderr parsl.dataflow.dflow DEBUG: Task 1478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1478 has memoization hash 124974eff54050b2427d18b5fd09612f parsl.dataflow.memoization INFO: Task 1478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1478 try 0 launched on executor multi with executor id 1479 parsl.dataflow.dflow INFO: Standard output for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/07141926-6f00-484c-ba4c-b6d33dddf06d_isrForCrosstalkSources_1031717_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/07141926-6f00-484c-ba4c-b6d33dddf06d_isrForCrosstalkSources_1031717_61.stderr parsl.dataflow.dflow DEBUG: Task 1479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1479 has memoization hash ba2d79396f5cfb3d7d733bc6d9a0e202 parsl.dataflow.memoization INFO: Task 1479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1479 try 0 launched on executor multi with executor id 1480 parsl.dataflow.dflow INFO: Standard output for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f3999668-55a4-40ed-b808-fc28ae52eb2f_isrForCrosstalkSources_1031699_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f3999668-55a4-40ed-b808-fc28ae52eb2f_isrForCrosstalkSources_1031699_29.stderr parsl.dataflow.dflow DEBUG: Task 1480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1480 has memoization hash f4ec53eef3cf040d18f89214093c285f parsl.dataflow.memoization INFO: Task 1480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1480 try 0 launched on executor multi with executor id 1481 parsl.dataflow.dflow INFO: Standard output for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/26835edb-4046-4d6f-94eb-09fd3d22c754_isrForCrosstalkSources_1031718_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/26835edb-4046-4d6f-94eb-09fd3d22c754_isrForCrosstalkSources_1031718_6.stderr parsl.dataflow.dflow DEBUG: Task 1481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1481 has memoization hash 27320fabff1fa44226469de5c29750aa parsl.dataflow.memoization INFO: Task 1481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1481 try 0 launched on executor multi with executor id 1482 parsl.dataflow.dflow INFO: Standard output for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f163cb32-de9e-425a-bc60-9eead08bbfc9_isrForCrosstalkSources_1031737_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f163cb32-de9e-425a-bc60-9eead08bbfc9_isrForCrosstalkSources_1031737_18.stderr parsl.dataflow.dflow DEBUG: Task 1482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1482 has memoization hash 69aa12b0609c367deea577537569b8d5 parsl.dataflow.memoization INFO: Task 1482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1482 try 0 launched on executor multi with executor id 1483 parsl.dataflow.dflow INFO: Standard output for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/bf152285-688e-42fd-ab4c-e7cc26621abf_isrForCrosstalkSources_1031701_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/bf152285-688e-42fd-ab4c-e7cc26621abf_isrForCrosstalkSources_1031701_15.stderr parsl.dataflow.dflow DEBUG: Task 1483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1483 has memoization hash d4d9384bb2f65f067ecbd97cc7640212 parsl.dataflow.memoization INFO: Task 1483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1483 try 0 launched on executor multi with executor id 1484 parsl.dataflow.dflow INFO: Standard output for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3221af40-4101-44fb-8b08-0bd594cc8af5_isrForCrosstalkSources_1031731_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3221af40-4101-44fb-8b08-0bd594cc8af5_isrForCrosstalkSources_1031731_17.stderr parsl.dataflow.dflow DEBUG: Task 1484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1484 has memoization hash f13356dadc435e759d8bddeeca796af5 parsl.dataflow.memoization INFO: Task 1484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1484 try 0 launched on executor multi with executor id 1485 parsl.dataflow.dflow INFO: Standard output for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1b0b5525-48c9-47f3-bbfa-9f5365d6333a_isrForCrosstalkSources_1031733_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1b0b5525-48c9-47f3-bbfa-9f5365d6333a_isrForCrosstalkSources_1031733_31.stderr parsl.dataflow.dflow DEBUG: Task 1485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1485 has memoization hash 1cbcb4478106b6e9bcb388e7cd943c18 parsl.dataflow.memoization INFO: Task 1485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1485 try 0 launched on executor multi with executor id 1486 parsl.dataflow.dflow INFO: Standard output for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/f4f6d9fd-4576-407a-a081-5eac434f0ed3_isrForCrosstalkSources_1031722_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/f4f6d9fd-4576-407a-a081-5eac434f0ed3_isrForCrosstalkSources_1031722_46.stderr parsl.dataflow.dflow DEBUG: Task 1486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1486 has memoization hash fb60b4e74c8a4ac51298fdc4a5954863 parsl.dataflow.memoization INFO: Task 1486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1486 try 0 launched on executor multi with executor id 1487 parsl.dataflow.dflow INFO: Standard output for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/7f83e4c9-a3bf-48ec-8de5-07b19541eb61_isrForCrosstalkSources_1031736_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/7f83e4c9-a3bf-48ec-8de5-07b19541eb61_isrForCrosstalkSources_1031736_46.stderr parsl.dataflow.dflow DEBUG: Task 1487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1487 has memoization hash 5557e74ce1089d3c1f8df26726fc8931 parsl.dataflow.memoization INFO: Task 1487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1487 try 0 launched on executor multi with executor id 1488 parsl.dataflow.dflow INFO: Standard output for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e63d0527-2812-4652-8aeb-9b6580c86023_isrForCrosstalkSources_1031736_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e63d0527-2812-4652-8aeb-9b6580c86023_isrForCrosstalkSources_1031736_29.stderr parsl.dataflow.dflow DEBUG: Task 1488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1488 has memoization hash 642174482e0b830211169dbd3c9aae1e parsl.dataflow.memoization INFO: Task 1488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1488 try 0 launched on executor multi with executor id 1489 parsl.dataflow.dflow INFO: Standard output for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/61d36a3c-451f-48ee-bbb6-b2d171776b64_isrForCrosstalkSources_1031723_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/61d36a3c-451f-48ee-bbb6-b2d171776b64_isrForCrosstalkSources_1031723_49.stderr parsl.dataflow.dflow DEBUG: Task 1489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1489 has memoization hash 4f00f89d3bcad24bcf4420c00e4a7f38 parsl.dataflow.memoization INFO: Task 1489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1489 try 0 launched on executor multi with executor id 1490 parsl.dataflow.dflow INFO: Standard output for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ac628fc8-ff61-428f-b10b-6d3d77173e4c_isrForCrosstalkSources_1031724_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ac628fc8-ff61-428f-b10b-6d3d77173e4c_isrForCrosstalkSources_1031724_39.stderr parsl.dataflow.dflow DEBUG: Task 1490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1490 has memoization hash c32301e9f3c581e0e41768912f4869a3 parsl.dataflow.memoization INFO: Task 1490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1490 try 0 launched on executor multi with executor id 1491 parsl.dataflow.dflow INFO: Standard output for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/f48c62f8-66fb-4507-94be-6f09a5a44a80_isrForCrosstalkSources_1031724_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/f48c62f8-66fb-4507-94be-6f09a5a44a80_isrForCrosstalkSources_1031724_52.stderr parsl.dataflow.dflow DEBUG: Task 1491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1491 has memoization hash 13e5025a6a16620819cb60ef8cf1402a parsl.dataflow.memoization INFO: Task 1491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1491 try 0 launched on executor multi with executor id 1492 parsl.dataflow.dflow INFO: Standard output for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/4e54b04c-a956-4678-a436-48627b58bf23_isrForCrosstalkSources_1031736_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/4e54b04c-a956-4678-a436-48627b58bf23_isrForCrosstalkSources_1031736_19.stderr parsl.dataflow.dflow DEBUG: Task 1492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1492 has memoization hash 00e3c623046ef223029ec98534fc2533 parsl.dataflow.memoization INFO: Task 1492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1492 try 0 launched on executor multi with executor id 1493 parsl.dataflow.dflow INFO: Standard output for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/6927d862-5195-4b98-ab22-2f5ca2b1d61c_isrForCrosstalkSources_1031727_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/6927d862-5195-4b98-ab22-2f5ca2b1d61c_isrForCrosstalkSources_1031727_11.stderr parsl.dataflow.dflow DEBUG: Task 1493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1493 has memoization hash cb0a53d4e5c426b2bd702bba020a941f parsl.dataflow.memoization INFO: Task 1493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1493 try 0 launched on executor multi with executor id 1494 parsl.dataflow.dflow INFO: Standard output for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/99d3a6d3-6343-421e-8e35-f313f9dd41e4_isrForCrosstalkSources_1031731_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/99d3a6d3-6343-421e-8e35-f313f9dd41e4_isrForCrosstalkSources_1031731_57.stderr parsl.dataflow.dflow DEBUG: Task 1494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1494 has memoization hash 149e7ec6fab59e120fd59dda017639f5 parsl.dataflow.memoization INFO: Task 1494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1494 try 0 launched on executor multi with executor id 1495 parsl.dataflow.dflow INFO: Standard output for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6a2551f3-d8e8-4a42-a906-1b552691c0cf_isrForCrosstalkSources_1031716_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6a2551f3-d8e8-4a42-a906-1b552691c0cf_isrForCrosstalkSources_1031716_61.stderr parsl.dataflow.dflow DEBUG: Task 1495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1495 has memoization hash 179cc75748892a296c093c183d496828 parsl.dataflow.memoization INFO: Task 1495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1495 try 0 launched on executor multi with executor id 1496 parsl.dataflow.dflow INFO: Standard output for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/535e482a-d707-401c-a5b3-81c53c42f204_isrForCrosstalkSources_1031714_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/535e482a-d707-401c-a5b3-81c53c42f204_isrForCrosstalkSources_1031714_5.stderr parsl.dataflow.dflow DEBUG: Task 1496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1496 has memoization hash 7be1b12c321502826395fc2a3c63597e parsl.dataflow.memoization INFO: Task 1496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1496 try 0 launched on executor multi with executor id 1497 parsl.dataflow.dflow INFO: Standard output for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/26af4dbd-2472-4fb1-b046-3166fa94756a_isrForCrosstalkSources_1031707_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/26af4dbd-2472-4fb1-b046-3166fa94756a_isrForCrosstalkSources_1031707_23.stderr parsl.dataflow.dflow DEBUG: Task 1497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1497 has memoization hash 18c501af1e6eb198e984462797c000ed parsl.dataflow.memoization INFO: Task 1497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1497 try 0 launched on executor multi with executor id 1498 parsl.dataflow.dflow INFO: Standard output for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0bc8f33e-d523-46a4-a870-576e5c603ddb_isrForCrosstalkSources_1031711_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0bc8f33e-d523-46a4-a870-576e5c603ddb_isrForCrosstalkSources_1031711_54.stderr parsl.dataflow.dflow DEBUG: Task 1498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1498 has memoization hash b0725c3ca580b51068a1fe8c259fbd07 parsl.dataflow.memoization INFO: Task 1498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1498 try 0 launched on executor multi with executor id 1499 parsl.dataflow.dflow INFO: Standard output for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/82ab0417-5eab-4f23-8a3f-cd32be7859c9_isrForCrosstalkSources_1031730_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/82ab0417-5eab-4f23-8a3f-cd32be7859c9_isrForCrosstalkSources_1031730_29.stderr parsl.dataflow.dflow DEBUG: Task 1499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1499 has memoization hash 7ce5dc02519317445a9aaeca335d34cc parsl.dataflow.memoization INFO: Task 1499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1499 try 0 launched on executor multi with executor id 1500 parsl.dataflow.dflow INFO: Standard output for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/520d09e9-17e0-4f67-8d06-d6bfdde317b5_isrForCrosstalkSources_1031702_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/520d09e9-17e0-4f67-8d06-d6bfdde317b5_isrForCrosstalkSources_1031702_8.stderr parsl.dataflow.dflow DEBUG: Task 1500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1500 has memoization hash 38c6bb89b7a2ebb885d9fc9847f31291 parsl.dataflow.memoization INFO: Task 1500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1500 try 0 launched on executor multi with executor id 1501 parsl.dataflow.dflow INFO: Standard output for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e923b42a-66db-4356-986f-41acd727d818_isrForCrosstalkSources_1031701_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e923b42a-66db-4356-986f-41acd727d818_isrForCrosstalkSources_1031701_4.stderr parsl.dataflow.dflow DEBUG: Task 1501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1501 has memoization hash c3435056f23f4003daaa319db47228af parsl.dataflow.memoization INFO: Task 1501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1501 try 0 launched on executor multi with executor id 1502 parsl.dataflow.dflow INFO: Standard output for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e65b9fb3-38f9-4bdf-a12d-8c2f98d4d2af_isrForCrosstalkSources_1031736_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e65b9fb3-38f9-4bdf-a12d-8c2f98d4d2af_isrForCrosstalkSources_1031736_14.stderr parsl.dataflow.dflow DEBUG: Task 1502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1502 has memoization hash af8566a993e1bb14527f423944dd3bd5 parsl.dataflow.memoization INFO: Task 1502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1502 try 0 launched on executor multi with executor id 1503 parsl.dataflow.dflow INFO: Standard output for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a74778ab-a184-4b90-b163-7799a09ab65d_isrForCrosstalkSources_1031711_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a74778ab-a184-4b90-b163-7799a09ab65d_isrForCrosstalkSources_1031711_26.stderr parsl.dataflow.dflow DEBUG: Task 1503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1503 has memoization hash 110fde2b358f7c85dbd27fcfba718f7a parsl.dataflow.memoization INFO: Task 1503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1503 try 0 launched on executor multi with executor id 1504 parsl.dataflow.dflow INFO: Standard output for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2e18785a-88b4-4f96-9fe1-c2b3a1ef764f_isrForCrosstalkSources_1031723_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2e18785a-88b4-4f96-9fe1-c2b3a1ef764f_isrForCrosstalkSources_1031723_48.stderr parsl.dataflow.dflow DEBUG: Task 1504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1504 has memoization hash 9ad67d74eab9c6ceb86020fe5852e2ce parsl.dataflow.memoization INFO: Task 1504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1504 try 0 launched on executor multi with executor id 1505 parsl.dataflow.dflow INFO: Standard output for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/43b1614f-d4e1-4362-9674-54c53fe757ee_isrForCrosstalkSources_1031734_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/43b1614f-d4e1-4362-9674-54c53fe757ee_isrForCrosstalkSources_1031734_57.stderr parsl.dataflow.dflow DEBUG: Task 1505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1505 has memoization hash aae63be389d4a130b932c77064c12bce parsl.dataflow.memoization INFO: Task 1505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1505 try 0 launched on executor multi with executor id 1506 parsl.dataflow.dflow INFO: Standard output for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/cd4440af-7734-40b8-a177-8c66090fed1c_isrForCrosstalkSources_1031724_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/cd4440af-7734-40b8-a177-8c66090fed1c_isrForCrosstalkSources_1031724_53.stderr parsl.dataflow.dflow DEBUG: Task 1506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1506 has memoization hash 66a947816396f4cfb2601fbc19444a67 parsl.dataflow.memoization INFO: Task 1506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1506 try 0 launched on executor multi with executor id 1507 parsl.dataflow.dflow INFO: Standard output for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1cacac24-bc66-456a-ad4e-04b82ab8e7be_isrForCrosstalkSources_1031709_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1cacac24-bc66-456a-ad4e-04b82ab8e7be_isrForCrosstalkSources_1031709_40.stderr parsl.dataflow.dflow DEBUG: Task 1507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1507 has memoization hash ca2bfad7038b942e541a9e7f33a79cdd parsl.dataflow.memoization INFO: Task 1507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1507 try 0 launched on executor multi with executor id 1508 parsl.dataflow.dflow INFO: Standard output for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/77a0d118-e842-4d3d-ac3f-c894cf172911_isrForCrosstalkSources_1031708_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/77a0d118-e842-4d3d-ac3f-c894cf172911_isrForCrosstalkSources_1031708_2.stderr parsl.dataflow.dflow DEBUG: Task 1508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1508 has memoization hash 4e17780d5e6c793f547cfd7cb12d8309 parsl.dataflow.memoization INFO: Task 1508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1508 try 0 launched on executor multi with executor id 1509 parsl.dataflow.dflow INFO: Standard output for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d78d4273-d8a1-4b37-a7c5-550b9f2ef92e_isrForCrosstalkSources_1031726_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d78d4273-d8a1-4b37-a7c5-550b9f2ef92e_isrForCrosstalkSources_1031726_47.stderr parsl.dataflow.dflow DEBUG: Task 1509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1509 has memoization hash 704a10a02628cdbe3144a6ac9d63789a parsl.dataflow.memoization INFO: Task 1509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1509 try 0 launched on executor multi with executor id 1510 parsl.dataflow.dflow INFO: Standard output for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8c9af191-f4e5-4e0e-957f-83eefe1a8295_isrForCrosstalkSources_1031700_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8c9af191-f4e5-4e0e-957f-83eefe1a8295_isrForCrosstalkSources_1031700_48.stderr parsl.dataflow.dflow DEBUG: Task 1510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1510 has memoization hash 666837833b3cbd70b0756c1cddfd0c3f parsl.dataflow.memoization INFO: Task 1510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1510 try 0 launched on executor multi with executor id 1511 parsl.dataflow.dflow INFO: Standard output for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/11d7635c-079b-4f5e-8b9f-75f7fb5bb838_isrForCrosstalkSources_1031706_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/11d7635c-079b-4f5e-8b9f-75f7fb5bb838_isrForCrosstalkSources_1031706_23.stderr parsl.dataflow.dflow DEBUG: Task 1511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1511 has memoization hash 90c86cd838a165a06470ec7af239f4f4 parsl.dataflow.memoization INFO: Task 1511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1511 try 0 launched on executor multi with executor id 1512 parsl.dataflow.dflow INFO: Standard output for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3a4844e5-9f66-4671-8292-b40eab3fbf9e_isrForCrosstalkSources_1031731_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3a4844e5-9f66-4671-8292-b40eab3fbf9e_isrForCrosstalkSources_1031731_30.stderr parsl.dataflow.dflow DEBUG: Task 1512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1512 has memoization hash 97dcf13eed71c9754bd26a54b6a410a4 parsl.dataflow.memoization INFO: Task 1512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1512 try 0 launched on executor multi with executor id 1513 parsl.dataflow.dflow INFO: Standard output for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/a59d1236-c5e9-4e30-9683-f5fa9a8b018c_isrForCrosstalkSources_1031737_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/a59d1236-c5e9-4e30-9683-f5fa9a8b018c_isrForCrosstalkSources_1031737_56.stderr parsl.dataflow.dflow DEBUG: Task 1513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1513 has memoization hash 6c1113079e91fcd2f96a332a6358a3f2 parsl.dataflow.memoization INFO: Task 1513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1513 try 0 launched on executor multi with executor id 1514 parsl.dataflow.dflow INFO: Standard output for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/403a7066-fe8d-426b-812a-3716b188bfd4_isrForCrosstalkSources_1031729_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/403a7066-fe8d-426b-812a-3716b188bfd4_isrForCrosstalkSources_1031729_1.stderr parsl.dataflow.dflow DEBUG: Task 1514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1514 has memoization hash 7cbaa782abc9597a32c6d27d95080cd2 parsl.dataflow.memoization INFO: Task 1514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1514 try 0 launched on executor multi with executor id 1515 parsl.dataflow.dflow INFO: Standard output for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/7339aed5-cff6-41b3-a6a8-20eadb824587_isrForCrosstalkSources_1031699_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/7339aed5-cff6-41b3-a6a8-20eadb824587_isrForCrosstalkSources_1031699_27.stderr parsl.dataflow.dflow DEBUG: Task 1515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1515 has memoization hash 1950f6c5b6e73a9d75201ebf53d8faf7 parsl.dataflow.memoization INFO: Task 1515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1515 try 0 launched on executor multi with executor id 1516 parsl.dataflow.dflow INFO: Standard output for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/dbe3aa3f-53af-4d0f-8dfd-31b9bf770a90_isrForCrosstalkSources_1031702_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/dbe3aa3f-53af-4d0f-8dfd-31b9bf770a90_isrForCrosstalkSources_1031702_22.stderr parsl.dataflow.dflow DEBUG: Task 1516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1516 has memoization hash 8e82a74f913d16c7ce9b5b63184157e8 parsl.dataflow.memoization INFO: Task 1516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1516 try 0 launched on executor multi with executor id 1517 parsl.dataflow.dflow INFO: Standard output for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/135a0652-f61f-4289-aa2f-fb2d8ed6def2_isrForCrosstalkSources_1031708_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/135a0652-f61f-4289-aa2f-fb2d8ed6def2_isrForCrosstalkSources_1031708_56.stderr parsl.dataflow.dflow DEBUG: Task 1517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1517 has memoization hash afcbce7f5733aad7bfc63a9423763ee1 parsl.dataflow.memoization INFO: Task 1517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1517 try 0 launched on executor multi with executor id 1518 parsl.dataflow.dflow INFO: Standard output for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/0f22c703-ccff-4b67-8561-7f29e1f59806_isrForCrosstalkSources_1031728_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/0f22c703-ccff-4b67-8561-7f29e1f59806_isrForCrosstalkSources_1031728_18.stderr parsl.dataflow.dflow DEBUG: Task 1518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1518 has memoization hash acde16198ec6b45afbf8deff62d26c68 parsl.dataflow.memoization INFO: Task 1518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1518 try 0 launched on executor multi with executor id 1519 parsl.dataflow.dflow INFO: Standard output for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f947840d-d6bd-4605-9cd9-54989176a2dc_isrForCrosstalkSources_1031727_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f947840d-d6bd-4605-9cd9-54989176a2dc_isrForCrosstalkSources_1031727_50.stderr parsl.dataflow.dflow DEBUG: Task 1519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1519 has memoization hash 51943d4af2e590dd2d8ea1647c3bb0ba parsl.dataflow.memoization INFO: Task 1519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1519 try 0 launched on executor multi with executor id 1520 parsl.dataflow.dflow INFO: Standard output for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/939d812c-bf84-4798-8271-92b61512965f_isrForCrosstalkSources_1031719_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/939d812c-bf84-4798-8271-92b61512965f_isrForCrosstalkSources_1031719_22.stderr parsl.dataflow.dflow DEBUG: Task 1520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1520 has memoization hash f6fe3ed5d6b20b392dd2b75564eafbef parsl.dataflow.memoization INFO: Task 1520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1520 try 0 launched on executor multi with executor id 1521 parsl.dataflow.dflow INFO: Standard output for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1788db7e-498b-4e33-80ca-bb7f29cbc1a7_isrForCrosstalkSources_1031703_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1788db7e-498b-4e33-80ca-bb7f29cbc1a7_isrForCrosstalkSources_1031703_60.stderr parsl.dataflow.dflow DEBUG: Task 1521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1521 has memoization hash 7ff9ed00baa3f7b2bfadd1fb5411a20a parsl.dataflow.memoization INFO: Task 1521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1521 try 0 launched on executor multi with executor id 1522 parsl.dataflow.dflow INFO: Standard output for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ece69d7f-d8cf-4582-8724-300cfa1286fe_isrForCrosstalkSources_1031715_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ece69d7f-d8cf-4582-8724-300cfa1286fe_isrForCrosstalkSources_1031715_6.stderr parsl.dataflow.dflow DEBUG: Task 1522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1522 has memoization hash c376cee28bb5972782c042b133c178aa parsl.dataflow.memoization INFO: Task 1522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1522 try 0 launched on executor multi with executor id 1523 parsl.dataflow.dflow INFO: Standard output for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/960b7339-9673-45cd-839d-ff4dee13a4c6_isrForCrosstalkSources_1031717_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/960b7339-9673-45cd-839d-ff4dee13a4c6_isrForCrosstalkSources_1031717_31.stderr parsl.dataflow.dflow DEBUG: Task 1523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1523 has memoization hash f084ecf8ee61eff4afc0f88c349255d4 parsl.dataflow.memoization INFO: Task 1523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1523 try 0 launched on executor multi with executor id 1524 parsl.dataflow.dflow INFO: Standard output for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/adbe1f5d-c051-4762-ad8f-7d03c24e1819_isrForCrosstalkSources_1031720_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/adbe1f5d-c051-4762-ad8f-7d03c24e1819_isrForCrosstalkSources_1031720_14.stderr parsl.dataflow.dflow DEBUG: Task 1524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1524 has memoization hash 81273630ff8ba2b2c7b7d798833cb035 parsl.dataflow.memoization INFO: Task 1524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1524 try 0 launched on executor multi with executor id 1525 parsl.dataflow.dflow INFO: Standard output for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1159bbaa-3504-47ba-9ca2-ce8d612b1582_isrForCrosstalkSources_1031735_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1159bbaa-3504-47ba-9ca2-ce8d612b1582_isrForCrosstalkSources_1031735_22.stderr parsl.dataflow.dflow DEBUG: Task 1525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1525 has memoization hash d72e798c46fe69fc239b844f6aad2029 parsl.dataflow.memoization INFO: Task 1525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1525 try 0 launched on executor multi with executor id 1526 parsl.dataflow.dflow INFO: Standard output for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/4895fd93-f0df-4313-9314-b40c583edab0_isrForCrosstalkSources_1031714_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/4895fd93-f0df-4313-9314-b40c583edab0_isrForCrosstalkSources_1031714_16.stderr parsl.dataflow.dflow DEBUG: Task 1526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1526 has memoization hash 7ec9f8d446fc81a1953aa471747ffe57 parsl.dataflow.memoization INFO: Task 1526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1526 try 0 launched on executor multi with executor id 1527 parsl.dataflow.dflow INFO: Standard output for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/35fcb2a1-44a8-4d66-9f68-77ecf3330824_isrForCrosstalkSources_1031705_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/35fcb2a1-44a8-4d66-9f68-77ecf3330824_isrForCrosstalkSources_1031705_15.stderr parsl.dataflow.dflow DEBUG: Task 1527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1527 has memoization hash 928db3e15fff594e516cd3e6a5ab371e parsl.dataflow.memoization INFO: Task 1527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1527 try 0 launched on executor multi with executor id 1528 parsl.dataflow.dflow INFO: Standard output for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/013681b6-b80a-49df-b6e5-a9aae18a846f_isrForCrosstalkSources_1031720_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/013681b6-b80a-49df-b6e5-a9aae18a846f_isrForCrosstalkSources_1031720_43.stderr parsl.dataflow.dflow DEBUG: Task 1528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1528 has memoization hash 56e09a399256cbddbdf6f188d1f5c018 parsl.dataflow.memoization INFO: Task 1528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1528 try 0 launched on executor multi with executor id 1529 parsl.dataflow.dflow INFO: Standard output for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5d551355-0dbf-4fb0-947e-5b23a6e1d5da_isrForCrosstalkSources_1031727_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5d551355-0dbf-4fb0-947e-5b23a6e1d5da_isrForCrosstalkSources_1031727_25.stderr parsl.dataflow.dflow DEBUG: Task 1529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1529 has memoization hash 2b0cb9b7661a3d644d086fc03f33c76f parsl.dataflow.memoization INFO: Task 1529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1529 try 0 launched on executor multi with executor id 1530 parsl.dataflow.dflow INFO: Standard output for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/2b7f4856-9bdc-4941-8ae6-2be2b761e961_isrForCrosstalkSources_1031706_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/2b7f4856-9bdc-4941-8ae6-2be2b761e961_isrForCrosstalkSources_1031706_38.stderr parsl.dataflow.dflow DEBUG: Task 1530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1530 has memoization hash f5d11ac39bce7803baa108f6f4aa12e5 parsl.dataflow.memoization INFO: Task 1530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1530 try 0 launched on executor multi with executor id 1531 parsl.dataflow.dflow INFO: Standard output for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/b6a68dee-3122-433f-be2e-51812e72f94e_isrForCrosstalkSources_1031722_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/b6a68dee-3122-433f-be2e-51812e72f94e_isrForCrosstalkSources_1031722_39.stderr parsl.dataflow.dflow DEBUG: Task 1531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1531 has memoization hash 90162d2be957d5aa2e1de994e2ee78b0 parsl.dataflow.memoization INFO: Task 1531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1531 try 0 launched on executor multi with executor id 1532 parsl.dataflow.dflow INFO: Standard output for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/59296da6-b6e4-4651-b94d-9b23d91080d6_isrForCrosstalkSources_1031700_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/59296da6-b6e4-4651-b94d-9b23d91080d6_isrForCrosstalkSources_1031700_24.stderr parsl.dataflow.dflow DEBUG: Task 1532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1532 has memoization hash 9eee8d61a2f8b51c134a06d9bce1070c parsl.dataflow.memoization INFO: Task 1532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1532 try 0 launched on executor multi with executor id 1533 parsl.dataflow.dflow INFO: Standard output for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1baae9f5-c5a4-45f6-ba2b-6af00c6603bd_isrForCrosstalkSources_1031715_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1baae9f5-c5a4-45f6-ba2b-6af00c6603bd_isrForCrosstalkSources_1031715_21.stderr parsl.dataflow.dflow DEBUG: Task 1533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1533 has memoization hash 9075a6803a9e54d1505fe30c5fb52bc4 parsl.dataflow.memoization INFO: Task 1533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1533 try 0 launched on executor multi with executor id 1534 parsl.dataflow.dflow INFO: Standard output for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/ff0d99a6-8fb4-41f9-a7ac-115695794376_isrForCrosstalkSources_1031718_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/ff0d99a6-8fb4-41f9-a7ac-115695794376_isrForCrosstalkSources_1031718_50.stderr parsl.dataflow.dflow DEBUG: Task 1534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1534 has memoization hash 11cd73ad363fba31cb2b21dec3cff689 parsl.dataflow.memoization INFO: Task 1534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1534 try 0 launched on executor multi with executor id 1535 parsl.dataflow.dflow INFO: Standard output for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/6c0aa16d-0f95-4faa-8a6c-2cc84264f616_isrForCrosstalkSources_1031725_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/6c0aa16d-0f95-4faa-8a6c-2cc84264f616_isrForCrosstalkSources_1031725_26.stderr parsl.dataflow.dflow DEBUG: Task 1535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1535 has memoization hash ff2c8a6f8620a04edd01706791a97760 parsl.dataflow.memoization INFO: Task 1535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1535 try 0 launched on executor multi with executor id 1536 parsl.dataflow.dflow INFO: Standard output for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/210d18a2-0ffe-447b-824e-31ce5f7fafa2_isrForCrosstalkSources_1031710_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/210d18a2-0ffe-447b-824e-31ce5f7fafa2_isrForCrosstalkSources_1031710_47.stderr parsl.dataflow.dflow DEBUG: Task 1536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1536 has memoization hash e7a626bed4500af410e6f9acb862f014 parsl.dataflow.memoization INFO: Task 1536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1536 try 0 launched on executor multi with executor id 1537 parsl.dataflow.dflow INFO: Standard output for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/30b2c9c2-1a09-4644-bcb8-5c523a49c156_isrForCrosstalkSources_1031716_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/30b2c9c2-1a09-4644-bcb8-5c523a49c156_isrForCrosstalkSources_1031716_20.stderr parsl.dataflow.dflow DEBUG: Task 1537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1537 has memoization hash 9e5ed4aa6eb49c15580412dd5f7dae1f parsl.dataflow.memoization INFO: Task 1537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1537 try 0 launched on executor multi with executor id 1538 parsl.dataflow.dflow INFO: Standard output for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/4b0ad832-fb26-409f-9c15-c4dd655cee7a_isrForCrosstalkSources_1031713_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/4b0ad832-fb26-409f-9c15-c4dd655cee7a_isrForCrosstalkSources_1031713_12.stderr parsl.dataflow.dflow DEBUG: Task 1538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1538 has memoization hash 5070536fade253a1af04b97bbe9d71f7 parsl.dataflow.memoization INFO: Task 1538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1538 try 0 launched on executor multi with executor id 1539 parsl.dataflow.dflow INFO: Standard output for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/809fe97f-387a-4cd0-b140-a9be3a4f2ce6_isrForCrosstalkSources_1031713_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/809fe97f-387a-4cd0-b140-a9be3a4f2ce6_isrForCrosstalkSources_1031713_56.stderr parsl.dataflow.dflow DEBUG: Task 1539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1539 has memoization hash 9da2bce2e9639284e222778e6e79c600 parsl.dataflow.memoization INFO: Task 1539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1539 try 0 launched on executor multi with executor id 1540 parsl.dataflow.dflow INFO: Standard output for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/9b460b07-fd63-4dce-ac1a-733c752064f0_isrForCrosstalkSources_1031734_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/9b460b07-fd63-4dce-ac1a-733c752064f0_isrForCrosstalkSources_1031734_58.stderr parsl.dataflow.dflow DEBUG: Task 1540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1540 has memoization hash 728c43fcfec46cd431fe49ca40101120 parsl.dataflow.memoization INFO: Task 1540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1540 try 0 launched on executor multi with executor id 1541 parsl.dataflow.dflow INFO: Standard output for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/575d2e01-eda0-4378-84e6-2e771a4237b5_isrForCrosstalkSources_1031706_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/575d2e01-eda0-4378-84e6-2e771a4237b5_isrForCrosstalkSources_1031706_51.stderr parsl.dataflow.dflow DEBUG: Task 1541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1541 has memoization hash b41868b11a90f3b0d1d020d2fdf6115a parsl.dataflow.memoization INFO: Task 1541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1541 try 0 launched on executor multi with executor id 1542 parsl.dataflow.dflow INFO: Standard output for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d9133e78-0dc3-4f08-a343-4c81a0a6db1a_isrForCrosstalkSources_1031701_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d9133e78-0dc3-4f08-a343-4c81a0a6db1a_isrForCrosstalkSources_1031701_36.stderr parsl.dataflow.dflow DEBUG: Task 1542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1542 has memoization hash 89432aa354802a0c5b5a42631bf93911 parsl.dataflow.memoization INFO: Task 1542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1542 try 0 launched on executor multi with executor id 1543 parsl.dataflow.dflow INFO: Standard output for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/efb3f015-224c-443f-a591-12521e63315d_isrForCrosstalkSources_1031705_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/efb3f015-224c-443f-a591-12521e63315d_isrForCrosstalkSources_1031705_47.stderr parsl.dataflow.dflow DEBUG: Task 1543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1543 has memoization hash ec2712d71fd4e1ceff9af65f251cbe4a parsl.dataflow.memoization INFO: Task 1543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1543 try 0 launched on executor multi with executor id 1544 parsl.dataflow.dflow INFO: Standard output for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/a30fca6c-91e7-475c-8886-8cf460d7b888_isrForCrosstalkSources_1031734_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/a30fca6c-91e7-475c-8886-8cf460d7b888_isrForCrosstalkSources_1031734_25.stderr parsl.dataflow.dflow DEBUG: Task 1544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1544 has memoization hash 7a2804937af00ed261dc3ac4a50064dd parsl.dataflow.memoization INFO: Task 1544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1544 try 0 launched on executor multi with executor id 1545 parsl.dataflow.dflow INFO: Standard output for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ac0995b0-e808-4b5b-9ae2-cb6aa0925897_isrForCrosstalkSources_1031726_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ac0995b0-e808-4b5b-9ae2-cb6aa0925897_isrForCrosstalkSources_1031726_50.stderr parsl.dataflow.dflow DEBUG: Task 1545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1545 has memoization hash 5fa5f0df45b1ff8a8904e7524db68508 parsl.dataflow.memoization INFO: Task 1545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1545 try 0 launched on executor multi with executor id 1546 parsl.dataflow.dflow INFO: Standard output for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b907b494-1fee-449f-bea0-e8d761472e77_isrForCrosstalkSources_1031713_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b907b494-1fee-449f-bea0-e8d761472e77_isrForCrosstalkSources_1031713_1.stderr parsl.dataflow.dflow DEBUG: Task 1546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1546 has memoization hash aa1837101d2358e5a44d4b968c63c72b parsl.dataflow.memoization INFO: Task 1546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1546 try 0 launched on executor multi with executor id 1547 parsl.dataflow.dflow INFO: Standard output for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e92d6e1d-6cd3-48ab-a815-59083a3d4e0e_isrForCrosstalkSources_1031712_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e92d6e1d-6cd3-48ab-a815-59083a3d4e0e_isrForCrosstalkSources_1031712_54.stderr parsl.dataflow.dflow DEBUG: Task 1547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1547 has memoization hash 085922a7f33a26f3d6d2df80b994a231 parsl.dataflow.memoization INFO: Task 1547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1547 try 0 launched on executor multi with executor id 1548 parsl.dataflow.dflow INFO: Standard output for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/90172052-4a96-4c17-b7d0-278faa3f12c4_isrForCrosstalkSources_1031709_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/90172052-4a96-4c17-b7d0-278faa3f12c4_isrForCrosstalkSources_1031709_38.stderr parsl.dataflow.dflow DEBUG: Task 1548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1548 has memoization hash 397eb72b0f3b788ab810cb992d02986c parsl.dataflow.memoization INFO: Task 1548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1548 try 0 launched on executor multi with executor id 1549 parsl.dataflow.dflow INFO: Standard output for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/1150f1c4-c624-46fb-ba69-d20ee4919bc3_isrForCrosstalkSources_1031726_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/1150f1c4-c624-46fb-ba69-d20ee4919bc3_isrForCrosstalkSources_1031726_40.stderr parsl.dataflow.dflow DEBUG: Task 1549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1549 has memoization hash 3bf1c796f5e0aa13a525e72d255c9b37 parsl.dataflow.memoization INFO: Task 1549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1549 try 0 launched on executor multi with executor id 1550 parsl.dataflow.dflow INFO: Standard output for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/598ec560-e8d4-4c59-b7cf-7d05900830c2_isrForCrosstalkSources_1031733_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/598ec560-e8d4-4c59-b7cf-7d05900830c2_isrForCrosstalkSources_1031733_54.stderr parsl.dataflow.dflow DEBUG: Task 1550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1550 has memoization hash 6bb039525665d6067196d542bc2bbb14 parsl.dataflow.memoization INFO: Task 1550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1550 try 0 launched on executor multi with executor id 1551 parsl.dataflow.dflow INFO: Standard output for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5b8ecf54-f919-4311-a826-0591fea57b07_isrForCrosstalkSources_1031730_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5b8ecf54-f919-4311-a826-0591fea57b07_isrForCrosstalkSources_1031730_54.stderr parsl.dataflow.dflow DEBUG: Task 1551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1551 has memoization hash 3a8f3ab426d1ed53ceeed488beab2cd2 parsl.dataflow.memoization INFO: Task 1551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1551 try 0 launched on executor multi with executor id 1552 parsl.dataflow.dflow INFO: Standard output for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c33ed61e-0225-4b63-a1d6-2ce22456b85f_isrForCrosstalkSources_1031729_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c33ed61e-0225-4b63-a1d6-2ce22456b85f_isrForCrosstalkSources_1031729_51.stderr parsl.dataflow.dflow DEBUG: Task 1552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1552 has memoization hash 391fda5f7015db426b5a70121426e5d6 parsl.dataflow.memoization INFO: Task 1552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1552 try 0 launched on executor multi with executor id 1553 parsl.dataflow.dflow INFO: Standard output for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/19a6821c-ad6c-43dc-826f-749b7833b4d2_isrForCrosstalkSources_1031734_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/19a6821c-ad6c-43dc-826f-749b7833b4d2_isrForCrosstalkSources_1031734_11.stderr parsl.dataflow.dflow DEBUG: Task 1553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1553 has memoization hash 3d2e00ffb3627820bdc50b25bbdcdced parsl.dataflow.memoization INFO: Task 1553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1553 try 0 launched on executor multi with executor id 1554 parsl.dataflow.dflow INFO: Standard output for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/fb5bf47c-b970-4f8a-bbae-cbbf15f7172b_isrForCrosstalkSources_1031728_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/fb5bf47c-b970-4f8a-bbae-cbbf15f7172b_isrForCrosstalkSources_1031728_59.stderr parsl.dataflow.dflow DEBUG: Task 1554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1554 has memoization hash d7edb8acff471c25da91d06fd43ef002 parsl.dataflow.memoization INFO: Task 1554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1554 try 0 launched on executor multi with executor id 1555 parsl.dataflow.dflow INFO: Standard output for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/456cfab5-7933-4d09-ba0f-fa9692656b62_isrForCrosstalkSources_1031714_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/456cfab5-7933-4d09-ba0f-fa9692656b62_isrForCrosstalkSources_1031714_53.stderr parsl.dataflow.dflow DEBUG: Task 1555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1555 has memoization hash 72801248a6b8925c9ccbc3934c076649 parsl.dataflow.memoization INFO: Task 1555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1555 try 0 launched on executor multi with executor id 1556 parsl.dataflow.dflow INFO: Standard output for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/79d05077-02ea-4f92-bf67-799c3a9b3710_isrForCrosstalkSources_1031706_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/79d05077-02ea-4f92-bf67-799c3a9b3710_isrForCrosstalkSources_1031706_15.stderr parsl.dataflow.dflow DEBUG: Task 1556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1556 has memoization hash 98dc3fd33b2f10fb29730e180afad048 parsl.dataflow.memoization INFO: Task 1556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1556 try 0 launched on executor multi with executor id 1557 parsl.dataflow.dflow INFO: Standard output for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ea9a4594-3615-4341-a9e5-99742539997e_isrForCrosstalkSources_1031725_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ea9a4594-3615-4341-a9e5-99742539997e_isrForCrosstalkSources_1031725_41.stderr parsl.dataflow.dflow DEBUG: Task 1557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1557 has memoization hash 41cfbb35237762ae0a0bdd9b96fec466 parsl.dataflow.memoization INFO: Task 1557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1557 try 0 launched on executor multi with executor id 1558 parsl.dataflow.dflow INFO: Standard output for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a06e77c2-5c03-4f5e-90c3-9d878c0078ae_isrForCrosstalkSources_1031721_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a06e77c2-5c03-4f5e-90c3-9d878c0078ae_isrForCrosstalkSources_1031721_33.stderr parsl.dataflow.dflow DEBUG: Task 1558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1558 has memoization hash 1512412fe917849ef3cf0b7bbe62963a parsl.dataflow.memoization INFO: Task 1558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1558 try 0 launched on executor multi with executor id 1559 parsl.dataflow.dflow INFO: Standard output for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bdcbb4b5-8645-494e-afd6-2c54c5f7e929_isrForCrosstalkSources_1031715_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bdcbb4b5-8645-494e-afd6-2c54c5f7e929_isrForCrosstalkSources_1031715_62.stderr parsl.dataflow.dflow DEBUG: Task 1559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1559 has memoization hash eac932a473509d62aaf8b8a29886b386 parsl.dataflow.memoization INFO: Task 1559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1559 try 0 launched on executor multi with executor id 1560 parsl.dataflow.dflow INFO: Standard output for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/257022d2-4d58-4b6a-a8d0-2b18db6f1870_isrForCrosstalkSources_1031709_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/257022d2-4d58-4b6a-a8d0-2b18db6f1870_isrForCrosstalkSources_1031709_7.stderr parsl.dataflow.dflow DEBUG: Task 1560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1560 has memoization hash 53d8957e0e96daf83a43aa3129436970 parsl.dataflow.memoization INFO: Task 1560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1560 try 0 launched on executor multi with executor id 1561 parsl.dataflow.dflow INFO: Standard output for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/18d1e226-817c-4380-830e-3059cac397ca_isrForCrosstalkSources_1031700_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/18d1e226-817c-4380-830e-3059cac397ca_isrForCrosstalkSources_1031700_38.stderr parsl.dataflow.dflow DEBUG: Task 1561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1561 has memoization hash 5b64aa0a5499c34502bb65b7a17fb2c4 parsl.dataflow.memoization INFO: Task 1561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1561 try 0 launched on executor multi with executor id 1562 parsl.dataflow.dflow INFO: Standard output for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9b437c32-0194-4bf9-8254-1d6a6ca533e9_isrForCrosstalkSources_1031716_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9b437c32-0194-4bf9-8254-1d6a6ca533e9_isrForCrosstalkSources_1031716_22.stderr parsl.dataflow.dflow DEBUG: Task 1562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1562 has memoization hash 15080d86dbc11486af25912dd8085506 parsl.dataflow.memoization INFO: Task 1562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1562 try 0 launched on executor multi with executor id 1563 parsl.dataflow.dflow INFO: Standard output for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8bed08f3-f3bb-43ef-b68d-07d5dd74743d_isrForCrosstalkSources_1031708_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8bed08f3-f3bb-43ef-b68d-07d5dd74743d_isrForCrosstalkSources_1031708_58.stderr parsl.dataflow.dflow DEBUG: Task 1563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1563 has memoization hash bcc5ba025a4cef40183d4a90d611402c parsl.dataflow.memoization INFO: Task 1563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1563 try 0 launched on executor multi with executor id 1564 parsl.dataflow.dflow INFO: Standard output for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/2c5c49af-7958-41d9-992d-cc91da50250a_isrForCrosstalkSources_1031733_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/2c5c49af-7958-41d9-992d-cc91da50250a_isrForCrosstalkSources_1031733_60.stderr parsl.dataflow.dflow DEBUG: Task 1564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1564 has memoization hash 48207b416ceb6fbb9417bd2bda9923b1 parsl.dataflow.memoization INFO: Task 1564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1564 try 0 launched on executor multi with executor id 1565 parsl.dataflow.dflow INFO: Standard output for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/bc694f37-1e14-4ba9-bf13-ca29db973667_isrForCrosstalkSources_1031702_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/bc694f37-1e14-4ba9-bf13-ca29db973667_isrForCrosstalkSources_1031702_18.stderr parsl.dataflow.dflow DEBUG: Task 1565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1565 has memoization hash 0fa89e6c7a081ba43894664e6778ce60 parsl.dataflow.memoization INFO: Task 1565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1565 try 0 launched on executor multi with executor id 1566 parsl.dataflow.dflow INFO: Standard output for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/377240ee-fdeb-42bd-ac31-36238560ddeb_isrForCrosstalkSources_1031717_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/377240ee-fdeb-42bd-ac31-36238560ddeb_isrForCrosstalkSources_1031717_27.stderr parsl.dataflow.dflow DEBUG: Task 1566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1566 has memoization hash b6abe49e56ae36c0723d7f35779225d3 parsl.dataflow.memoization INFO: Task 1566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1566 try 0 launched on executor multi with executor id 1567 parsl.dataflow.dflow INFO: Standard output for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/01df4bf8-8390-4bfb-a97e-ac0d08ae740b_isrForCrosstalkSources_1031710_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/01df4bf8-8390-4bfb-a97e-ac0d08ae740b_isrForCrosstalkSources_1031710_17.stderr parsl.dataflow.dflow DEBUG: Task 1567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1567 has memoization hash 094dc660eb5137a6cb7eec7ae69debe8 parsl.dataflow.memoization INFO: Task 1567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1567 try 0 launched on executor multi with executor id 1568 parsl.dataflow.dflow INFO: Standard output for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/ebe633ac-47d1-4de1-9db4-8f5f445344a5_isrForCrosstalkSources_1031721_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/ebe633ac-47d1-4de1-9db4-8f5f445344a5_isrForCrosstalkSources_1031721_44.stderr parsl.dataflow.dflow DEBUG: Task 1568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1568 has memoization hash 786947cba9e3ead8dd17cff9ac9cf904 parsl.dataflow.memoization INFO: Task 1568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1568 try 0 launched on executor multi with executor id 1569 parsl.dataflow.dflow INFO: Standard output for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a117b22f-ee31-41a4-ae8d-5532dd6f46a0_isrForCrosstalkSources_1031731_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a117b22f-ee31-41a4-ae8d-5532dd6f46a0_isrForCrosstalkSources_1031731_12.stderr parsl.dataflow.dflow DEBUG: Task 1569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1569 has memoization hash f86991ab80e44355b8943fc2c6698eae parsl.dataflow.memoization INFO: Task 1569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1569 try 0 launched on executor multi with executor id 1570 parsl.dataflow.dflow INFO: Standard output for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2f1971d5-fefe-4773-aa69-b936bde7f845_isrForCrosstalkSources_1031734_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2f1971d5-fefe-4773-aa69-b936bde7f845_isrForCrosstalkSources_1031734_22.stderr parsl.dataflow.dflow DEBUG: Task 1570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1570 has memoization hash a0d40a2cbfc796b5f71e779e21e14aeb parsl.dataflow.memoization INFO: Task 1570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1570 try 0 launched on executor multi with executor id 1571 parsl.dataflow.dflow INFO: Standard output for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/428b2cec-2717-485f-864e-171b1cfa90e5_isrForCrosstalkSources_1031728_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/428b2cec-2717-485f-864e-171b1cfa90e5_isrForCrosstalkSources_1031728_32.stderr parsl.dataflow.dflow DEBUG: Task 1571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1571 has memoization hash 6e8894c4cf2bcbc7ab43b5b202629b8a parsl.dataflow.memoization INFO: Task 1571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1571 try 0 launched on executor multi with executor id 1572 parsl.dataflow.dflow INFO: Standard output for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/44d7266c-30bc-4952-8d4d-fb67f1806952_isrForCrosstalkSources_1031708_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/44d7266c-30bc-4952-8d4d-fb67f1806952_isrForCrosstalkSources_1031708_30.stderr parsl.dataflow.dflow DEBUG: Task 1572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1572 has memoization hash 26ebd2d985d35a369eb3af8667b9d549 parsl.dataflow.memoization INFO: Task 1572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1572 try 0 launched on executor multi with executor id 1573 parsl.dataflow.dflow INFO: Standard output for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8c85fab3-4a8c-40da-adf3-db8ee199ad03_isrForCrosstalkSources_1031721_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8c85fab3-4a8c-40da-adf3-db8ee199ad03_isrForCrosstalkSources_1031721_36.stderr parsl.dataflow.dflow DEBUG: Task 1573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1573 has memoization hash 4d202ab83e41093cc0e4ef36f8abc080 parsl.dataflow.memoization INFO: Task 1573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1573 try 0 launched on executor multi with executor id 1574 parsl.dataflow.dflow INFO: Standard output for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fca500ee-19e6-4f85-999f-16ad3304f797_isrForCrosstalkSources_1031707_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fca500ee-19e6-4f85-999f-16ad3304f797_isrForCrosstalkSources_1031707_58.stderr parsl.dataflow.dflow DEBUG: Task 1574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1574 has memoization hash 8dd4d59a9978d919c625d11d5fb34e18 parsl.dataflow.memoization INFO: Task 1574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1574 try 0 launched on executor multi with executor id 1575 parsl.dataflow.dflow INFO: Standard output for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/e8948ac1-4fbe-4e55-aa3d-3d52d376a11d_isrForCrosstalkSources_1031713_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/e8948ac1-4fbe-4e55-aa3d-3d52d376a11d_isrForCrosstalkSources_1031713_57.stderr parsl.dataflow.dflow DEBUG: Task 1575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1575 has memoization hash a2343df2d178038dde46ce35acb98b88 parsl.dataflow.memoization INFO: Task 1575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1575 try 0 launched on executor multi with executor id 1576 parsl.dataflow.dflow INFO: Standard output for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/e3261476-9ed2-4694-84bb-6dc6395fff2a_isrForCrosstalkSources_1031714_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/e3261476-9ed2-4694-84bb-6dc6395fff2a_isrForCrosstalkSources_1031714_24.stderr parsl.dataflow.dflow DEBUG: Task 1576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1576 has memoization hash 57c2d079ee874ee52a0bfa7f105252c9 parsl.dataflow.memoization INFO: Task 1576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1576 try 0 launched on executor multi with executor id 1577 parsl.dataflow.dflow INFO: Standard output for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/653cfeab-a6bc-4cbc-be9d-d692a8c4ae08_isrForCrosstalkSources_1031705_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/653cfeab-a6bc-4cbc-be9d-d692a8c4ae08_isrForCrosstalkSources_1031705_54.stderr parsl.dataflow.dflow DEBUG: Task 1577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1577 has memoization hash 5de3236a04ed228f21c3f56a4a24fa74 parsl.dataflow.memoization INFO: Task 1577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1577 try 0 launched on executor multi with executor id 1578 parsl.dataflow.dflow INFO: Standard output for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/3b71408a-28ff-450d-9e2e-f14ddefbab91_isrForCrosstalkSources_1031726_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/3b71408a-28ff-450d-9e2e-f14ddefbab91_isrForCrosstalkSources_1031726_18.stderr parsl.dataflow.dflow DEBUG: Task 1578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1578 has memoization hash ee54ab59deb5ed4bf18be5050eadfe83 parsl.dataflow.memoization INFO: Task 1578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1578 try 0 launched on executor multi with executor id 1579 parsl.dataflow.dflow INFO: Standard output for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/d529e2fb-a748-462c-8bc1-054972944274_isrForCrosstalkSources_1031713_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/d529e2fb-a748-462c-8bc1-054972944274_isrForCrosstalkSources_1031713_46.stderr parsl.dataflow.dflow DEBUG: Task 1579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1579 has memoization hash 0fcaff7d298fcc5c9cddf2deb92fe597 parsl.dataflow.memoization INFO: Task 1579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1579 try 0 launched on executor multi with executor id 1580 parsl.dataflow.dflow INFO: Standard output for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fdfd91b5-2c8d-4c09-966d-0b0b3cd51383_isrForCrosstalkSources_1031723_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fdfd91b5-2c8d-4c09-966d-0b0b3cd51383_isrForCrosstalkSources_1031723_16.stderr parsl.dataflow.dflow DEBUG: Task 1580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1580 has memoization hash 3e71af292b006a27dade0b426dda5c44 parsl.dataflow.memoization INFO: Task 1580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1580 try 0 launched on executor multi with executor id 1581 parsl.dataflow.dflow INFO: Standard output for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/ecdab980-80bf-4bcb-9b0d-0386887b7581_isrForCrosstalkSources_1031732_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/ecdab980-80bf-4bcb-9b0d-0386887b7581_isrForCrosstalkSources_1031732_6.stderr parsl.dataflow.dflow DEBUG: Task 1581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1581 has memoization hash 811afbc7afe4e2c115de096946cf3822 parsl.dataflow.memoization INFO: Task 1581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1581 try 0 launched on executor multi with executor id 1582 parsl.dataflow.dflow INFO: Standard output for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/aaebfb9f-393e-449d-a399-767b60daef04_isrForCrosstalkSources_1031714_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/aaebfb9f-393e-449d-a399-767b60daef04_isrForCrosstalkSources_1031714_20.stderr parsl.dataflow.dflow DEBUG: Task 1582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1582 has memoization hash c094e65dbf79206e38e10c9a3ad66389 parsl.dataflow.memoization INFO: Task 1582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2faa2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1582 try 0 launched on executor multi with executor id 1583 parsl.dataflow.dflow INFO: Standard output for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2af11f85-0743-495c-a0dd-7aaaeebf692e_isrForCrosstalkSources_1031727_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2af11f85-0743-495c-a0dd-7aaaeebf692e_isrForCrosstalkSources_1031727_47.stderr parsl.dataflow.dflow DEBUG: Task 1583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1583 has memoization hash c38062165b2d717ad80390f1b2eee03a parsl.dataflow.memoization INFO: Task 1583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1583 try 0 launched on executor multi with executor id 1584 parsl.dataflow.dflow INFO: Standard output for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/139b6b1f-6197-49f1-9ec5-1b8a803ea48e_isrForCrosstalkSources_1031732_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/139b6b1f-6197-49f1-9ec5-1b8a803ea48e_isrForCrosstalkSources_1031732_29.stderr parsl.dataflow.dflow DEBUG: Task 1584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1584 has memoization hash f9936a464135e81f1d99b4b53a2f0200 parsl.dataflow.memoization INFO: Task 1584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1584 try 0 launched on executor multi with executor id 1585 parsl.dataflow.dflow INFO: Standard output for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8d38e5c2-fc67-4191-98d9-e55fbbd7277b_isrForCrosstalkSources_1031708_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8d38e5c2-fc67-4191-98d9-e55fbbd7277b_isrForCrosstalkSources_1031708_16.stderr parsl.dataflow.dflow DEBUG: Task 1585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1585 has memoization hash 740cc3b4f56656141e2d36632e9fca6d parsl.dataflow.memoization INFO: Task 1585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1585 try 0 launched on executor multi with executor id 1586 parsl.dataflow.dflow INFO: Standard output for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/167778b3-ec1c-4ad3-885b-499faa5fdd91_isrForCrosstalkSources_1031717_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/167778b3-ec1c-4ad3-885b-499faa5fdd91_isrForCrosstalkSources_1031717_52.stderr parsl.dataflow.dflow DEBUG: Task 1586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1586 has memoization hash 7dd754c1349a3e34d059a91748a13395 parsl.dataflow.memoization INFO: Task 1586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9482c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1586 try 0 launched on executor multi with executor id 1587 parsl.dataflow.dflow INFO: Standard output for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b7822bca-9659-44b7-8e60-14df1547ee4c_isrForCrosstalkSources_1031707_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b7822bca-9659-44b7-8e60-14df1547ee4c_isrForCrosstalkSources_1031707_20.stderr parsl.dataflow.dflow DEBUG: Task 1587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1587 has memoization hash a374829fb4ca690c2bb95853238d8b5a parsl.dataflow.memoization INFO: Task 1587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1587 try 0 launched on executor multi with executor id 1588 parsl.dataflow.dflow INFO: Standard output for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/38d2e4a8-e407-4752-aa5f-cd105b567997_isrForCrosstalkSources_1031718_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/38d2e4a8-e407-4752-aa5f-cd105b567997_isrForCrosstalkSources_1031718_60.stderr parsl.dataflow.dflow DEBUG: Task 1588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1588 has memoization hash f119477645c7350797d44bc7bce135ae parsl.dataflow.memoization INFO: Task 1588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1588 try 0 launched on executor multi with executor id 1589 parsl.dataflow.dflow INFO: Standard output for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/85b6bd53-aac1-4fe3-91e1-360a538774fe_isrForCrosstalkSources_1031703_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/85b6bd53-aac1-4fe3-91e1-360a538774fe_isrForCrosstalkSources_1031703_51.stderr parsl.dataflow.dflow DEBUG: Task 1589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1589 has memoization hash 6a0d3a4242588c460daa218925c08993 parsl.dataflow.memoization INFO: Task 1589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9484a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1589 try 0 launched on executor multi with executor id 1590 parsl.dataflow.dflow INFO: Standard output for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7a4a8190-8281-40e5-9019-50c2bc95ecf5_isrForCrosstalkSources_1031704_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7a4a8190-8281-40e5-9019-50c2bc95ecf5_isrForCrosstalkSources_1031704_55.stderr parsl.dataflow.dflow DEBUG: Task 1590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1590 has memoization hash f29f0032001a803668cb3b7c7c33cd05 parsl.dataflow.memoization INFO: Task 1590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1590 try 0 launched on executor multi with executor id 1591 parsl.dataflow.dflow INFO: Standard output for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/65da1710-8b12-49de-aa6b-d5193cd4c536_isrForCrosstalkSources_1031715_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/65da1710-8b12-49de-aa6b-d5193cd4c536_isrForCrosstalkSources_1031715_61.stderr parsl.dataflow.dflow DEBUG: Task 1591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1591 has memoization hash 641e706488211fbe4d5580a032b84baf parsl.dataflow.memoization INFO: Task 1591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9485e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1591 try 0 launched on executor multi with executor id 1592 parsl.dataflow.dflow INFO: Standard output for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3110feba-8386-4901-85fc-be244574ec2d_isrForCrosstalkSources_1031713_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3110feba-8386-4901-85fc-be244574ec2d_isrForCrosstalkSources_1031713_13.stderr parsl.dataflow.dflow DEBUG: Task 1592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1592 has memoization hash 9501c5a1d7ec14e72588bd4e77dfb937 parsl.dataflow.memoization INFO: Task 1592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1592 try 0 launched on executor multi with executor id 1593 parsl.dataflow.dflow INFO: Standard output for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/11c4cb86-47a0-4c60-9447-bca87e3cb14a_isrForCrosstalkSources_1031714_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/11c4cb86-47a0-4c60-9447-bca87e3cb14a_isrForCrosstalkSources_1031714_51.stderr parsl.dataflow.dflow DEBUG: Task 1593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1593 has memoization hash d7b2cf51a2863e4d6414fc3bf3f03f8c parsl.dataflow.memoization INFO: Task 1593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1593 try 0 launched on executor multi with executor id 1594 parsl.dataflow.dflow INFO: Standard output for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/63d9d0d7-e7b1-4024-818c-b5cde3bdd317_isrForCrosstalkSources_1031703_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/63d9d0d7-e7b1-4024-818c-b5cde3bdd317_isrForCrosstalkSources_1031703_9.stderr parsl.dataflow.dflow DEBUG: Task 1594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1594 has memoization hash 1c34ccb79fc27de31d915cf4aad414c5 parsl.dataflow.memoization INFO: Task 1594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9487c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1594 try 0 launched on executor multi with executor id 1595 parsl.dataflow.dflow INFO: Standard output for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/a3e91b86-5eba-4c93-97f1-6c56672e6f90_isrForCrosstalkSources_1031700_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/a3e91b86-5eba-4c93-97f1-6c56672e6f90_isrForCrosstalkSources_1031700_46.stderr parsl.dataflow.dflow DEBUG: Task 1595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1595 has memoization hash a372cde2a77e4d12c395d5a116a29327 parsl.dataflow.memoization INFO: Task 1595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1595 try 0 launched on executor multi with executor id 1596 parsl.dataflow.dflow INFO: Standard output for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/ab1d8340-b193-4358-b10f-c61865941964_isrForCrosstalkSources_1031719_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/ab1d8340-b193-4358-b10f-c61865941964_isrForCrosstalkSources_1031719_14.stderr parsl.dataflow.dflow DEBUG: Task 1596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1596 has memoization hash b4db58338be3e7efc0b02684e2f481e3 parsl.dataflow.memoization INFO: Task 1596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1596 try 0 launched on executor multi with executor id 1597 parsl.dataflow.dflow INFO: Standard output for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1fa8f7e9-afde-497e-8355-55a74bd0c85c_isrForCrosstalkSources_1031715_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1fa8f7e9-afde-497e-8355-55a74bd0c85c_isrForCrosstalkSources_1031715_44.stderr parsl.dataflow.dflow DEBUG: Task 1597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1597 has memoization hash 6426de473558f6c40f975a5a2d3a7eb7 parsl.dataflow.memoization INFO: Task 1597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:16 proc_lsst.multi:170 [INFO] [multi] job_id 17147952 proc_lsst.multi INFO: [multi] job_id 17147952 2024-03-17 04:02:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-17 04:02:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147952 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147952 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17147952 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-17 04:02:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-17 04:02:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-17 04:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1597 try 0 launched on executor multi with executor id 1598 parsl.dataflow.dflow INFO: Standard output for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d7eb2f56-e061-4f56-8278-31cb3ead396f_isrForCrosstalkSources_1031722_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d7eb2f56-e061-4f56-8278-31cb3ead396f_isrForCrosstalkSources_1031722_44.stderr parsl.dataflow.dflow DEBUG: Task 1598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1598 has memoization hash ee33ace8f871a798d17b3a2647292980 parsl.dataflow.memoization INFO: Task 1598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa028e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1598 try 0 launched on executor multi with executor id 1599 parsl.dataflow.dflow INFO: Standard output for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/c434acb4-dee3-43cb-bb06-5f8d449255cc_isrForCrosstalkSources_1031717_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/c434acb4-dee3-43cb-bb06-5f8d449255cc_isrForCrosstalkSources_1031717_47.stderr parsl.dataflow.dflow DEBUG: Task 1599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1599 has memoization hash c0ca308e230add832084c224663c9053 parsl.dataflow.memoization INFO: Task 1599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1599 try 0 launched on executor multi with executor id 1600 parsl.dataflow.dflow INFO: Standard output for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f7683daf-4483-4b0e-bcb5-4a28605a5bac_isrForCrosstalkSources_1031703_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f7683daf-4483-4b0e-bcb5-4a28605a5bac_isrForCrosstalkSources_1031703_62.stderr parsl.dataflow.dflow DEBUG: Task 1600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1600 has memoization hash 67f5358be6dac4c264eaab964392c796 parsl.dataflow.memoization INFO: Task 1600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa039c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1600 try 0 launched on executor multi with executor id 1601 parsl.dataflow.dflow INFO: Standard output for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/355f11da-ebc5-4eb0-a638-da0c2a20dece_isrForCrosstalkSources_1031725_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/355f11da-ebc5-4eb0-a638-da0c2a20dece_isrForCrosstalkSources_1031725_24.stderr parsl.dataflow.dflow DEBUG: Task 1601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1601 has memoization hash 90b0d1d8e0326544168b59d20a24b4d1 parsl.dataflow.memoization INFO: Task 1601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9489a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1601 try 0 launched on executor multi with executor id 1602 parsl.dataflow.dflow INFO: Standard output for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/97f4d939-722d-4093-85ba-cf7999964f9f_isrForCrosstalkSources_1031700_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/97f4d939-722d-4093-85ba-cf7999964f9f_isrForCrosstalkSources_1031700_62.stderr parsl.dataflow.dflow DEBUG: Task 1602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1602 has memoization hash 044e9b68e753487496ff76755f74c314 parsl.dataflow.memoization INFO: Task 1602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1602 try 0 launched on executor multi with executor id 1603 parsl.dataflow.dflow INFO: Standard output for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/315aeffb-5760-400b-9933-d54d07a0f3f9_isrForCrosstalkSources_1031738_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/315aeffb-5760-400b-9933-d54d07a0f3f9_isrForCrosstalkSources_1031738_48.stderr parsl.dataflow.dflow DEBUG: Task 1603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1603 has memoization hash ee69c2dd87794949c1a4a72e4cdab789 parsl.dataflow.memoization INFO: Task 1603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1603 try 0 launched on executor multi with executor id 1604 parsl.dataflow.dflow INFO: Standard output for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/218dffb0-a6a1-4263-964e-b7a579d2613e_isrForCrosstalkSources_1031724_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/218dffb0-a6a1-4263-964e-b7a579d2613e_isrForCrosstalkSources_1031724_60.stderr parsl.dataflow.dflow DEBUG: Task 1604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1604 has memoization hash 139fed699ba95c5387411d29b927cd0c parsl.dataflow.memoization INFO: Task 1604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1604 try 0 launched on executor multi with executor id 1605 parsl.dataflow.dflow INFO: Standard output for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/165ccdc3-efa9-4e6d-8546-c337fcaee0f0_isrForCrosstalkSources_1031702_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/165ccdc3-efa9-4e6d-8546-c337fcaee0f0_isrForCrosstalkSources_1031702_16.stderr parsl.dataflow.dflow DEBUG: Task 1605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1605 has memoization hash abc5819fd2081d9f6cbb8ff2e364bddf parsl.dataflow.memoization INFO: Task 1605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1605 try 0 launched on executor multi with executor id 1606 parsl.dataflow.dflow INFO: Standard output for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/5c986052-af36-47c5-a067-de77cbaef626_isrForCrosstalkSources_1031723_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/5c986052-af36-47c5-a067-de77cbaef626_isrForCrosstalkSources_1031723_39.stderr parsl.dataflow.dflow DEBUG: Task 1606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1606 has memoization hash b90e39aa85dc19db74a70eeb81b8ad58 parsl.dataflow.memoization INFO: Task 1606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1606 try 0 launched on executor multi with executor id 1607 parsl.dataflow.dflow INFO: Standard output for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/95e3352a-8c19-4ede-b688-13d844aafd55_isrForCrosstalkSources_1031736_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/95e3352a-8c19-4ede-b688-13d844aafd55_isrForCrosstalkSources_1031736_23.stderr parsl.dataflow.dflow DEBUG: Task 1607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1607 has memoization hash a60bf4d21e97ecb588972dcd38c891d9 parsl.dataflow.memoization INFO: Task 1607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1607 try 0 launched on executor multi with executor id 1608 parsl.dataflow.dflow INFO: Standard output for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/debb2906-8456-4b3c-8277-2ae044a53a56_isrForCrosstalkSources_1031706_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/debb2906-8456-4b3c-8277-2ae044a53a56_isrForCrosstalkSources_1031706_54.stderr parsl.dataflow.dflow DEBUG: Task 1608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1608 has memoization hash 4f05bae2e3da50ca94b792907852320a parsl.dataflow.memoization INFO: Task 1608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1608 try 0 launched on executor multi with executor id 1609 parsl.dataflow.dflow INFO: Standard output for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/05468c1e-1b84-4f19-b55d-83a1a493dba7_isrForCrosstalkSources_1031701_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/05468c1e-1b84-4f19-b55d-83a1a493dba7_isrForCrosstalkSources_1031701_24.stderr parsl.dataflow.dflow DEBUG: Task 1609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1609 has memoization hash 8a31645cde6aa82bd5dbddcd37866c84 parsl.dataflow.memoization INFO: Task 1609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1609 try 0 launched on executor multi with executor id 1610 parsl.dataflow.dflow INFO: Standard output for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ab56e0a7-f44c-4fa3-baeb-7cd0fa645735_isrForCrosstalkSources_1031731_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ab56e0a7-f44c-4fa3-baeb-7cd0fa645735_isrForCrosstalkSources_1031731_14.stderr parsl.dataflow.dflow DEBUG: Task 1610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1610 has memoization hash 23525a964641b65cbd205fe7c0f41423 parsl.dataflow.memoization INFO: Task 1610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1610 try 0 launched on executor multi with executor id 1611 parsl.dataflow.dflow INFO: Standard output for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/01503cc5-1c10-4c1f-874c-69c265571382_isrForCrosstalkSources_1031707_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/01503cc5-1c10-4c1f-874c-69c265571382_isrForCrosstalkSources_1031707_27.stderr parsl.dataflow.dflow DEBUG: Task 1611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1611 has memoization hash 18f7e9f460d7773350784a3a2707908e parsl.dataflow.memoization INFO: Task 1611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1611 try 0 launched on executor multi with executor id 1612 parsl.dataflow.dflow INFO: Standard output for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8ea9c130-b6f0-4408-938c-eddc9f13b420_isrForCrosstalkSources_1031729_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8ea9c130-b6f0-4408-938c-eddc9f13b420_isrForCrosstalkSources_1031729_16.stderr parsl.dataflow.dflow DEBUG: Task 1612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1612 has memoization hash adf9e1f7ffda09b83901dba8e3b55e1b parsl.dataflow.memoization INFO: Task 1612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9491c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1612 try 0 launched on executor multi with executor id 1613 parsl.dataflow.dflow INFO: Standard output for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/efcd86b3-10bf-4be6-a28f-6d97871af519_isrForCrosstalkSources_1031699_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/efcd86b3-10bf-4be6-a28f-6d97871af519_isrForCrosstalkSources_1031699_38.stderr parsl.dataflow.dflow DEBUG: Task 1613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1613 has memoization hash 0b8e314a4327891aecc4ae9e994ed494 parsl.dataflow.memoization INFO: Task 1613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1613 try 0 launched on executor multi with executor id 1614 parsl.dataflow.dflow INFO: Standard output for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/bf82ef4b-4dfa-4fc8-9f07-141fe0e5b61d_isrForCrosstalkSources_1031719_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/bf82ef4b-4dfa-4fc8-9f07-141fe0e5b61d_isrForCrosstalkSources_1031719_62.stderr parsl.dataflow.dflow DEBUG: Task 1614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1614 has memoization hash c4142742a14a9dc6bb2174d9791c19bf parsl.dataflow.memoization INFO: Task 1614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1614 try 0 launched on executor multi with executor id 1615 parsl.dataflow.dflow INFO: Standard output for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/721cd5d5-167f-43a7-b05d-7fd49c5c9739_isrForCrosstalkSources_1031702_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/721cd5d5-167f-43a7-b05d-7fd49c5c9739_isrForCrosstalkSources_1031702_5.stderr parsl.dataflow.dflow DEBUG: Task 1615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1615 has memoization hash 9047c7d8e0bd6c26f8857f0bd70105ec parsl.dataflow.memoization INFO: Task 1615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1615 try 0 launched on executor multi with executor id 1616 parsl.dataflow.dflow INFO: Standard output for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/30129a4c-5f41-465e-a8f3-51e5868e8193_isrForCrosstalkSources_1031724_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/30129a4c-5f41-465e-a8f3-51e5868e8193_isrForCrosstalkSources_1031724_62.stderr parsl.dataflow.dflow DEBUG: Task 1616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1616 has memoization hash ed3ac04393dc611ef376300f29f4ce88 parsl.dataflow.memoization INFO: Task 1616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1616 try 0 launched on executor multi with executor id 1617 parsl.dataflow.dflow INFO: Standard output for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/931b58ce-97ad-4670-a694-82ec2d0a80ee_isrForCrosstalkSources_1031735_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/931b58ce-97ad-4670-a694-82ec2d0a80ee_isrForCrosstalkSources_1031735_21.stderr parsl.dataflow.dflow DEBUG: Task 1617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1617 has memoization hash c7d8e6a8ded6aa2c3d4ec3038b292e57 parsl.dataflow.memoization INFO: Task 1617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9494e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1617 try 0 launched on executor multi with executor id 1618 parsl.dataflow.dflow INFO: Standard output for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/fe13a148-4aa4-48b7-acae-fccd94c80e9e_isrForCrosstalkSources_1031706_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/fe13a148-4aa4-48b7-acae-fccd94c80e9e_isrForCrosstalkSources_1031706_41.stderr parsl.dataflow.dflow DEBUG: Task 1618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1618 has memoization hash 8b76a7a65ce9fc10aa197e463a754e46 parsl.dataflow.memoization INFO: Task 1618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9493a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1618 try 0 launched on executor multi with executor id 1619 parsl.dataflow.dflow INFO: Standard output for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/f76da089-51b8-4a3e-adc6-43c990b58269_isrForCrosstalkSources_1031715_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/f76da089-51b8-4a3e-adc6-43c990b58269_isrForCrosstalkSources_1031715_14.stderr parsl.dataflow.dflow DEBUG: Task 1619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1619 has memoization hash 0baba2a7801cb0eeaf00bb8daa315213 parsl.dataflow.memoization INFO: Task 1619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1619 try 0 launched on executor multi with executor id 1620 parsl.dataflow.dflow INFO: Standard output for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c6959664-2b7e-476b-841a-da6f75a3b910_isrForCrosstalkSources_1031738_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c6959664-2b7e-476b-841a-da6f75a3b910_isrForCrosstalkSources_1031738_53.stderr parsl.dataflow.dflow DEBUG: Task 1620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1620 has memoization hash e12a96c9bf0495f1d9309d41c8a40d7f parsl.dataflow.memoization INFO: Task 1620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1620 try 0 launched on executor multi with executor id 1621 parsl.dataflow.dflow INFO: Standard output for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/9cb60c84-1494-49eb-99cf-99df2793052c_isrForCrosstalkSources_1031728_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/9cb60c84-1494-49eb-99cf-99df2793052c_isrForCrosstalkSources_1031728_8.stderr parsl.dataflow.dflow DEBUG: Task 1621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1621 has memoization hash 639b957b92e2a12e3adc9745b48d56bd parsl.dataflow.memoization INFO: Task 1621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1621 try 0 launched on executor multi with executor id 1622 parsl.dataflow.dflow INFO: Standard output for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8c1a3e0a-95bc-4520-8547-a0df5474eb8a_isrForCrosstalkSources_1031729_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8c1a3e0a-95bc-4520-8547-a0df5474eb8a_isrForCrosstalkSources_1031729_25.stderr parsl.dataflow.dflow DEBUG: Task 1622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1622 has memoization hash 89e28b30075bb2a54221c86c76397e13 parsl.dataflow.memoization INFO: Task 1622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1622 try 0 launched on executor multi with executor id 1623 parsl.dataflow.dflow INFO: Standard output for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8ca7793c-534c-47d7-9cb0-4226fb4575d2_isrForCrosstalkSources_1031707_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8ca7793c-534c-47d7-9cb0-4226fb4575d2_isrForCrosstalkSources_1031707_26.stderr parsl.dataflow.dflow DEBUG: Task 1623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1623 has memoization hash 165aef1c8ed8831e8b85d8dc995e8283 parsl.dataflow.memoization INFO: Task 1623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9496c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1623 try 0 launched on executor multi with executor id 1624 parsl.dataflow.dflow INFO: Standard output for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/acc1af7a-b64a-4963-8e65-5d21b1d5696c_isrForCrosstalkSources_1031707_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/acc1af7a-b64a-4963-8e65-5d21b1d5696c_isrForCrosstalkSources_1031707_4.stderr parsl.dataflow.dflow DEBUG: Task 1624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1624 has memoization hash eb7d74cad41a999631430172a745500a parsl.dataflow.memoization INFO: Task 1624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9498a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1624 try 0 launched on executor multi with executor id 1625 parsl.dataflow.dflow INFO: Standard output for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1029e10a-dd08-40f6-a1af-201a6483ecfa_isrForCrosstalkSources_1031732_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1029e10a-dd08-40f6-a1af-201a6483ecfa_isrForCrosstalkSources_1031732_4.stderr parsl.dataflow.dflow DEBUG: Task 1625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1625 has memoization hash fc5e93c9e9cae12faec3dce910d8ad1c parsl.dataflow.memoization INFO: Task 1625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1625 try 0 launched on executor multi with executor id 1626 parsl.dataflow.dflow INFO: Standard output for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/2352a158-8fbd-485d-8d66-3e5752e851c0_isrForCrosstalkSources_1031720_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/2352a158-8fbd-485d-8d66-3e5752e851c0_isrForCrosstalkSources_1031720_59.stderr parsl.dataflow.dflow DEBUG: Task 1626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1626 has memoization hash 3f03153529810575b7e2d7873f5683cb parsl.dataflow.memoization INFO: Task 1626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9499e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1626 try 0 launched on executor multi with executor id 1627 parsl.dataflow.dflow INFO: Standard output for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/7f85936d-6536-4baf-9e96-be35d823971a_isrForCrosstalkSources_1031726_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/7f85936d-6536-4baf-9e96-be35d823971a_isrForCrosstalkSources_1031726_27.stderr parsl.dataflow.dflow DEBUG: Task 1627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1627 has memoization hash a5131553fe5d1f7063039807d9bb78c0 parsl.dataflow.memoization INFO: Task 1627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1627 try 0 launched on executor multi with executor id 1628 parsl.dataflow.dflow INFO: Standard output for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/40f24cd4-9694-424e-8c35-8d269772c232_isrForCrosstalkSources_1031715_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/40f24cd4-9694-424e-8c35-8d269772c232_isrForCrosstalkSources_1031715_50.stderr parsl.dataflow.dflow DEBUG: Task 1628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1628 has memoization hash bdc52990f0b667bd37b2cc6d14f110f1 parsl.dataflow.memoization INFO: Task 1628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1628 try 0 launched on executor multi with executor id 1629 parsl.dataflow.dflow INFO: Standard output for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0c106321-bdc2-476f-b29e-a77122025938_isrForCrosstalkSources_1031714_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0c106321-bdc2-476f-b29e-a77122025938_isrForCrosstalkSources_1031714_40.stderr parsl.dataflow.dflow DEBUG: Task 1629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1629 has memoization hash c72269d4ad5c199ba59bec036731a559 parsl.dataflow.memoization INFO: Task 1629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1629 try 0 launched on executor multi with executor id 1630 parsl.dataflow.dflow INFO: Standard output for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/43369357-1171-4f1d-986f-de8a14835bbc_isrForCrosstalkSources_1031718_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/43369357-1171-4f1d-986f-de8a14835bbc_isrForCrosstalkSources_1031718_10.stderr parsl.dataflow.dflow DEBUG: Task 1630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1630 has memoization hash fb9d1adfdfbc159309f608bb7d73a3b9 parsl.dataflow.memoization INFO: Task 1630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1630 try 0 launched on executor multi with executor id 1631 parsl.dataflow.dflow INFO: Standard output for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/13c82374-8a4a-448b-af9b-866986579058_isrForCrosstalkSources_1031707_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/13c82374-8a4a-448b-af9b-866986579058_isrForCrosstalkSources_1031707_41.stderr parsl.dataflow.dflow DEBUG: Task 1631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1631 has memoization hash a9da319d8abfcf148575e00a14d19596 parsl.dataflow.memoization INFO: Task 1631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1631 try 0 launched on executor multi with executor id 1632 parsl.dataflow.dflow INFO: Standard output for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/97cdf895-34c4-423d-87cd-82343cfc22aa_isrForCrosstalkSources_1031729_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/97cdf895-34c4-423d-87cd-82343cfc22aa_isrForCrosstalkSources_1031729_23.stderr parsl.dataflow.dflow DEBUG: Task 1632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1632 has memoization hash 88a1bed341a3581129e46d375f854af3 parsl.dataflow.memoization INFO: Task 1632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1632 try 0 launched on executor multi with executor id 1633 parsl.dataflow.dflow INFO: Standard output for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/4e97a336-9b60-4d73-94e4-11c6e1f2690e_isrForCrosstalkSources_1031726_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/4e97a336-9b60-4d73-94e4-11c6e1f2690e_isrForCrosstalkSources_1031726_6.stderr parsl.dataflow.dflow DEBUG: Task 1633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1633 has memoization hash 2e731e9240abf8ccc51fd1604deab620 parsl.dataflow.memoization INFO: Task 1633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1633 try 0 launched on executor multi with executor id 1634 parsl.dataflow.dflow INFO: Standard output for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/fb8848ba-c06d-4134-99ac-7058c88baadc_isrForCrosstalkSources_1031733_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/fb8848ba-c06d-4134-99ac-7058c88baadc_isrForCrosstalkSources_1031733_33.stderr parsl.dataflow.dflow DEBUG: Task 1634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1634 has memoization hash 13ee6ff3975ab4f4a53a09db9f144405 parsl.dataflow.memoization INFO: Task 1634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1634 try 0 launched on executor multi with executor id 1635 parsl.dataflow.dflow INFO: Standard output for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8b7cb6a0-23f9-4aca-9ccd-06caad33be52_isrForCrosstalkSources_1031729_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8b7cb6a0-23f9-4aca-9ccd-06caad33be52_isrForCrosstalkSources_1031729_28.stderr parsl.dataflow.dflow DEBUG: Task 1635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1635 has memoization hash 28b4fa38860bd78e2b5f295ae63dc954 parsl.dataflow.memoization INFO: Task 1635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1635 try 0 launched on executor multi with executor id 1636 parsl.dataflow.dflow INFO: Standard output for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5bf7ee2f-1e04-4b2c-a85e-2e123cdc3b50_isrForCrosstalkSources_1031711_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5bf7ee2f-1e04-4b2c-a85e-2e123cdc3b50_isrForCrosstalkSources_1031711_13.stderr parsl.dataflow.dflow DEBUG: Task 1636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1636 has memoization hash 86adb7158dd1c92d6196169aeb823719 parsl.dataflow.memoization INFO: Task 1636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1636 try 0 launched on executor multi with executor id 1637 parsl.dataflow.dflow INFO: Standard output for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/3fa6feaf-2d3e-472b-8af4-0a4dd86edb64_isrForCrosstalkSources_1031727_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/3fa6feaf-2d3e-472b-8af4-0a4dd86edb64_isrForCrosstalkSources_1031727_36.stderr parsl.dataflow.dflow DEBUG: Task 1637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1637 has memoization hash 5f4a7c7d3f86f9136dec8c196abec117 parsl.dataflow.memoization INFO: Task 1637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1637 try 0 launched on executor multi with executor id 1638 parsl.dataflow.dflow INFO: Standard output for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/1e6ddc95-fbaf-4fcd-b97e-b656937d7661_isrForCrosstalkSources_1031720_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/1e6ddc95-fbaf-4fcd-b97e-b656937d7661_isrForCrosstalkSources_1031720_37.stderr parsl.dataflow.dflow DEBUG: Task 1638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1638 has memoization hash 9eaeecbee5512e80af8cb94992ee75e2 parsl.dataflow.memoization INFO: Task 1638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f949f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1638 try 0 launched on executor multi with executor id 1639 parsl.dataflow.dflow INFO: Standard output for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/250c38d1-173f-49e6-84f9-bafc8a314599_isrForCrosstalkSources_1031705_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/250c38d1-173f-49e6-84f9-bafc8a314599_isrForCrosstalkSources_1031705_60.stderr parsl.dataflow.dflow DEBUG: Task 1639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1639 has memoization hash 76e8e794e817f1565307b29f64c834ae parsl.dataflow.memoization INFO: Task 1639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1639 try 0 launched on executor multi with executor id 1640 parsl.dataflow.dflow INFO: Standard output for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ef38f34d-c398-412b-8839-f1ca76886b49_isrForCrosstalkSources_1031699_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ef38f34d-c398-412b-8839-f1ca76886b49_isrForCrosstalkSources_1031699_1.stderr parsl.dataflow.dflow DEBUG: Task 1640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1640 has memoization hash 8843eef4a5fe576c3971010b3281baf5 parsl.dataflow.memoization INFO: Task 1640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1640 try 0 launched on executor multi with executor id 1641 parsl.dataflow.dflow INFO: Standard output for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f37e385b-7fe8-4ea3-93f8-0835cf5d1f8a_isrForCrosstalkSources_1031730_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f37e385b-7fe8-4ea3-93f8-0835cf5d1f8a_isrForCrosstalkSources_1031730_16.stderr parsl.dataflow.dflow DEBUG: Task 1641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1641 has memoization hash 1c5c2df130b6d11f0db0ebcc4dddbcb4 parsl.dataflow.memoization INFO: Task 1641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1641 try 0 launched on executor multi with executor id 1642 parsl.dataflow.dflow INFO: Standard output for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3b58273c-42bb-4a13-9e37-b262cb61043c_isrForCrosstalkSources_1031723_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3b58273c-42bb-4a13-9e37-b262cb61043c_isrForCrosstalkSources_1031723_6.stderr parsl.dataflow.dflow DEBUG: Task 1642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1642 has memoization hash 8583de7e9a17ef6000f8aaa5cb2e96be parsl.dataflow.memoization INFO: Task 1642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1642 try 0 launched on executor multi with executor id 1643 parsl.dataflow.dflow INFO: Standard output for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/5a1b896f-9c0c-417b-a6a3-cdbf31a248f5_isrForCrosstalkSources_1031707_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/5a1b896f-9c0c-417b-a6a3-cdbf31a248f5_isrForCrosstalkSources_1031707_39.stderr parsl.dataflow.dflow DEBUG: Task 1643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1643 has memoization hash 0660b155d0547e055168a9c140eb829e parsl.dataflow.memoization INFO: Task 1643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1643 try 0 launched on executor multi with executor id 1644 parsl.dataflow.dflow INFO: Standard output for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/fd1bc628-4332-4885-8e88-d8decebfb07d_isrForCrosstalkSources_1031728_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/fd1bc628-4332-4885-8e88-d8decebfb07d_isrForCrosstalkSources_1031728_51.stderr parsl.dataflow.dflow DEBUG: Task 1644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1644 has memoization hash 1470af92f1aff73868cd38e1fb4da520 parsl.dataflow.memoization INFO: Task 1644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1644 try 0 launched on executor multi with executor id 1645 parsl.dataflow.dflow INFO: Standard output for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b4d04676-b11e-448a-83a8-8e0f41801068_isrForCrosstalkSources_1031718_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b4d04676-b11e-448a-83a8-8e0f41801068_isrForCrosstalkSources_1031718_58.stderr parsl.dataflow.dflow DEBUG: Task 1645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1645 has memoization hash 695698ac990c400f870ad161f84804b3 parsl.dataflow.memoization INFO: Task 1645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1645 try 0 launched on executor multi with executor id 1646 parsl.dataflow.dflow INFO: Standard output for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5b41415a-0d89-40be-a789-a843bdfc6ee1_isrForCrosstalkSources_1031736_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5b41415a-0d89-40be-a789-a843bdfc6ee1_isrForCrosstalkSources_1031736_45.stderr parsl.dataflow.dflow DEBUG: Task 1646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1646 has memoization hash 5fbe689984e75cc983e2edb62d2d5ff3 parsl.dataflow.memoization INFO: Task 1646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1646 try 0 launched on executor multi with executor id 1647 parsl.dataflow.dflow INFO: Standard output for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/be370e8d-aae8-4faf-84b7-3e69539d374d_isrForCrosstalkSources_1031708_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/be370e8d-aae8-4faf-84b7-3e69539d374d_isrForCrosstalkSources_1031708_27.stderr parsl.dataflow.dflow DEBUG: Task 1647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1647 has memoization hash a26dc7b1dff16531b5f0c79fde5fdb93 parsl.dataflow.memoization INFO: Task 1647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1647 try 0 launched on executor multi with executor id 1648 parsl.dataflow.dflow INFO: Standard output for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/e9302642-e9ba-461d-acbb-45e9406863d3_isrForCrosstalkSources_1031722_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/e9302642-e9ba-461d-acbb-45e9406863d3_isrForCrosstalkSources_1031722_61.stderr parsl.dataflow.dflow DEBUG: Task 1648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1648 has memoization hash 2205e5abcb07edee57baa883f3ed6623 parsl.dataflow.memoization INFO: Task 1648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1648 try 0 launched on executor multi with executor id 1649 parsl.dataflow.dflow INFO: Standard output for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/01cf5685-80a0-4a1f-9b0c-f425c875c404_isrForCrosstalkSources_1031718_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/01cf5685-80a0-4a1f-9b0c-f425c875c404_isrForCrosstalkSources_1031718_49.stderr parsl.dataflow.dflow DEBUG: Task 1649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1649 has memoization hash 7a4ccdea5a142e5a0b93ef911185c557 parsl.dataflow.memoization INFO: Task 1649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1649 try 0 launched on executor multi with executor id 1650 parsl.dataflow.dflow INFO: Standard output for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/93956866-2ac1-4cbc-a042-d6d99bdbcefa_isrForCrosstalkSources_1031708_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/93956866-2ac1-4cbc-a042-d6d99bdbcefa_isrForCrosstalkSources_1031708_19.stderr parsl.dataflow.dflow DEBUG: Task 1650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1650 has memoization hash bc027dadf8a8dd552a51a1e68add0348 parsl.dataflow.memoization INFO: Task 1650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1650 try 0 launched on executor multi with executor id 1651 parsl.dataflow.dflow INFO: Standard output for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/21e01752-26bf-4f4d-8165-99785b3c9a94_isrForCrosstalkSources_1031731_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/21e01752-26bf-4f4d-8165-99785b3c9a94_isrForCrosstalkSources_1031731_43.stderr parsl.dataflow.dflow DEBUG: Task 1651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1651 has memoization hash 8b65513f7a9d2725ab6821281beffafd parsl.dataflow.memoization INFO: Task 1651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1651 try 0 launched on executor multi with executor id 1652 parsl.dataflow.dflow INFO: Standard output for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b6b99375-8d37-4f94-936e-b25424d49838_isrForCrosstalkSources_1031716_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b6b99375-8d37-4f94-936e-b25424d49838_isrForCrosstalkSources_1031716_44.stderr parsl.dataflow.dflow DEBUG: Task 1652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1652 has memoization hash 5d080879110d2379de47422fa12b7cf6 parsl.dataflow.memoization INFO: Task 1652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1652 try 0 launched on executor multi with executor id 1653 parsl.dataflow.dflow INFO: Standard output for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/3fd6a7f5-667a-4479-848c-2ed51b99b147_isrForCrosstalkSources_1031724_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/3fd6a7f5-667a-4479-848c-2ed51b99b147_isrForCrosstalkSources_1031724_36.stderr parsl.dataflow.dflow DEBUG: Task 1653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1653 has memoization hash b9f3f260be947ac9bc75e86084d0d6a6 parsl.dataflow.memoization INFO: Task 1653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1653 try 0 launched on executor multi with executor id 1654 parsl.dataflow.dflow INFO: Standard output for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/8d434c01-3304-4c89-a974-052fedc8d0c0_isrForCrosstalkSources_1031722_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/8d434c01-3304-4c89-a974-052fedc8d0c0_isrForCrosstalkSources_1031722_20.stderr parsl.dataflow.dflow DEBUG: Task 1654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1654 has memoization hash 28db65cc070aa484c42304daf35e5638 parsl.dataflow.memoization INFO: Task 1654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1654 try 0 launched on executor multi with executor id 1655 parsl.dataflow.dflow INFO: Standard output for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3476349c-d8a0-454a-8e71-a0df66680ad2_isrForCrosstalkSources_1031738_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3476349c-d8a0-454a-8e71-a0df66680ad2_isrForCrosstalkSources_1031738_44.stderr parsl.dataflow.dflow DEBUG: Task 1655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1655 has memoization hash a3d7caf0354f42cd7bc73b6f8fd8274d parsl.dataflow.memoization INFO: Task 1655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1655 try 0 launched on executor multi with executor id 1656 parsl.dataflow.dflow INFO: Standard output for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0846cd2c-7bb9-4040-8ea0-bbfe238a1337_isrForCrosstalkSources_1031732_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0846cd2c-7bb9-4040-8ea0-bbfe238a1337_isrForCrosstalkSources_1031732_37.stderr parsl.dataflow.dflow DEBUG: Task 1656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1656 has memoization hash c2be6dfc7d5ae9fa6e724a8f22d4b7ef parsl.dataflow.memoization INFO: Task 1656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1656 try 0 launched on executor multi with executor id 1657 parsl.dataflow.dflow INFO: Standard output for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/3f4e8938-4688-409c-a86e-b2d1e4895049_isrForCrosstalkSources_1031715_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/3f4e8938-4688-409c-a86e-b2d1e4895049_isrForCrosstalkSources_1031715_37.stderr parsl.dataflow.dflow DEBUG: Task 1657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1657 has memoization hash a93e6e22e5913d435766f82d4bba9923 parsl.dataflow.memoization INFO: Task 1657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1657 try 0 launched on executor multi with executor id 1658 parsl.dataflow.dflow INFO: Standard output for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/22ea7043-0edf-4128-9133-9d4dbc44cbf5_isrForCrosstalkSources_1031737_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/22ea7043-0edf-4128-9133-9d4dbc44cbf5_isrForCrosstalkSources_1031737_61.stderr parsl.dataflow.dflow DEBUG: Task 1658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1658 has memoization hash 6a9d8e101dcc74d951952e44986a355c parsl.dataflow.memoization INFO: Task 1658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1658 try 0 launched on executor multi with executor id 1659 parsl.dataflow.dflow INFO: Standard output for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/f0b7e38f-171e-4201-98d3-f9e9958004fd_isrForCrosstalkSources_1031736_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/f0b7e38f-171e-4201-98d3-f9e9958004fd_isrForCrosstalkSources_1031736_38.stderr parsl.dataflow.dflow DEBUG: Task 1659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1659 has memoization hash 10797f4609760cf764b625ba5356056a parsl.dataflow.memoization INFO: Task 1659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1659 try 0 launched on executor multi with executor id 1660 parsl.dataflow.dflow INFO: Standard output for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/015cdbab-0c64-41ac-a4c8-fbca861347e8_isrForCrosstalkSources_1031702_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/015cdbab-0c64-41ac-a4c8-fbca861347e8_isrForCrosstalkSources_1031702_26.stderr parsl.dataflow.dflow DEBUG: Task 1660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1660 has memoization hash b5d3591a09ded86b152c19dc69f8df8c parsl.dataflow.memoization INFO: Task 1660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1660 try 0 launched on executor multi with executor id 1661 parsl.dataflow.dflow INFO: Standard output for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a35ee0ad-c351-41c9-86c9-075c4852a6db_isrForCrosstalkSources_1031712_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a35ee0ad-c351-41c9-86c9-075c4852a6db_isrForCrosstalkSources_1031712_26.stderr parsl.dataflow.dflow DEBUG: Task 1661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1661 has memoization hash 65cc9cca37515ee5df0c4e12f2289e58 parsl.dataflow.memoization INFO: Task 1661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1661 try 0 launched on executor multi with executor id 1662 parsl.dataflow.dflow INFO: Standard output for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0c69dd18-5fe2-43c8-867f-f1afe07f770d_isrForCrosstalkSources_1031721_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0c69dd18-5fe2-43c8-867f-f1afe07f770d_isrForCrosstalkSources_1031721_62.stderr parsl.dataflow.dflow DEBUG: Task 1662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1662 has memoization hash c60d3871011926cc741d7bc2c724f954 parsl.dataflow.memoization INFO: Task 1662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1662 try 0 launched on executor multi with executor id 1663 parsl.dataflow.dflow INFO: Standard output for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/da0a194f-c964-4ef7-9c08-1072da913d63_isrForCrosstalkSources_1031734_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/da0a194f-c964-4ef7-9c08-1072da913d63_isrForCrosstalkSources_1031734_8.stderr parsl.dataflow.dflow DEBUG: Task 1663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1663 has memoization hash 508f34e63c994cedffc221e60e10a2c5 parsl.dataflow.memoization INFO: Task 1663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1663 try 0 launched on executor multi with executor id 1664 parsl.dataflow.dflow INFO: Standard output for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/d24fa311-e22b-41eb-bf43-7403f92645c8_isrForCrosstalkSources_1031720_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/d24fa311-e22b-41eb-bf43-7403f92645c8_isrForCrosstalkSources_1031720_30.stderr parsl.dataflow.dflow DEBUG: Task 1664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1664 has memoization hash f4a0ca3aa43f0654d096b86c3935f2b5 parsl.dataflow.memoization INFO: Task 1664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1664 try 0 launched on executor multi with executor id 1665 parsl.dataflow.dflow INFO: Standard output for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/75221928-b092-4d6b-9a3f-ad132fa289e1_isrForCrosstalkSources_1031713_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/75221928-b092-4d6b-9a3f-ad132fa289e1_isrForCrosstalkSources_1031713_32.stderr parsl.dataflow.dflow DEBUG: Task 1665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1665 has memoization hash e8f7fd177c3036c25302cb587f69d125 parsl.dataflow.memoization INFO: Task 1665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1665 try 0 launched on executor multi with executor id 1666 parsl.dataflow.dflow INFO: Standard output for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/196fe2ae-762c-4158-bd7e-e076ce1d8a90_isrForCrosstalkSources_1031720_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/196fe2ae-762c-4158-bd7e-e076ce1d8a90_isrForCrosstalkSources_1031720_47.stderr parsl.dataflow.dflow DEBUG: Task 1666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1666 has memoization hash e366dcaf6b7729930c844a6c2900e35f parsl.dataflow.memoization INFO: Task 1666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1666 try 0 launched on executor multi with executor id 1667 parsl.dataflow.dflow INFO: Standard output for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/da929f66-7228-433a-9ef0-8df9e6bddb59_isrForCrosstalkSources_1031736_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/da929f66-7228-433a-9ef0-8df9e6bddb59_isrForCrosstalkSources_1031736_32.stderr parsl.dataflow.dflow DEBUG: Task 1667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1667 has memoization hash eecd7325a84bb3dfb9f1b5d6fd9bcf45 parsl.dataflow.memoization INFO: Task 1667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1667 try 0 launched on executor multi with executor id 1668 parsl.dataflow.dflow INFO: Standard output for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ead96e9d-f0a6-433c-90f0-cce7ce803b54_isrForCrosstalkSources_1031713_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ead96e9d-f0a6-433c-90f0-cce7ce803b54_isrForCrosstalkSources_1031713_40.stderr parsl.dataflow.dflow DEBUG: Task 1668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1668 has memoization hash e582013edc7ba53b24ba5b99db91bd99 parsl.dataflow.memoization INFO: Task 1668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3f20a457979e0d004ab807500bfe91f with result from task 1 parsl.dataflow.dflow INFO: Parsl task 1668 try 0 launched on executor multi with executor id 1669 parsl.dataflow.dflow INFO: Standard output for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d1379b56-77cf-48ed-9b4a-da764feaadd4_isrForCrosstalkSources_1031726_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d1379b56-77cf-48ed-9b4a-da764feaadd4_isrForCrosstalkSources_1031726_13.stderr parsl.dataflow.dflow DEBUG: Task 1669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1669 has memoization hash f158fa11670f52135802559f8c5f25d8 parsl.dataflow.memoization INFO: Task 1669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Parsl task 1669 try 0 launched on executor multi with executor id 1670 parsl.dataflow.dflow INFO: Standard output for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/10d21cf5-2579-498c-95e4-9ccb33572913_isrForCrosstalkSources_1031714_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/10d21cf5-2579-498c-95e4-9ccb33572913_isrForCrosstalkSources_1031714_18.stderr parsl.dataflow.dflow DEBUG: Task 1670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1670 has memoization hash fde4dfe66255169dd7cf47b6361eb5f3 parsl.dataflow.memoization INFO: Task 1670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/fcc22e3c-300f-464d-a0b9-c62d83c4ad57_isrForCrosstalkSources_1031714_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/fcc22e3c-300f-464d-a0b9-c62d83c4ad57_isrForCrosstalkSources_1031714_39.stderr parsl.dataflow.dflow INFO: Parsl task 1670 try 0 launched on executor multi with executor id 1671 parsl.dataflow.dflow INFO: Standard output for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/ece996fb-9292-4eb2-8489-cd368b623fad_isrForCrosstalkSources_1031734_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/ece996fb-9292-4eb2-8489-cd368b623fad_isrForCrosstalkSources_1031734_31.stderr parsl.dataflow.dflow DEBUG: Task 1671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1671 has memoization hash 6153696434add7a8c250f5e99fad70d0 parsl.dataflow.memoization INFO: Task 1671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1671 try 0 launched on executor multi with executor id 1672 parsl.dataflow.dflow INFO: Standard output for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/460b8c4c-558d-443d-af27-c3dafdf5f384_isrForCrosstalkSources_1031699_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/460b8c4c-558d-443d-af27-c3dafdf5f384_isrForCrosstalkSources_1031699_10.stderr parsl.dataflow.dflow DEBUG: Task 1672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1672 has memoization hash 7aa09f8e07452086e4dad137b38dc453 parsl.dataflow.memoization INFO: Task 1672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1672 try 0 launched on executor multi with executor id 1673 parsl.dataflow.dflow INFO: Standard output for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/88750516-5048-4257-bb8e-841ea8a5c2a5_isrForCrosstalkSources_1031732_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/88750516-5048-4257-bb8e-841ea8a5c2a5_isrForCrosstalkSources_1031732_26.stderr parsl.dataflow.dflow DEBUG: Task 1673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1673 has memoization hash 1ffed08838cdf5965956bb7c6be1bf9e parsl.dataflow.memoization INFO: Task 1673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1673 try 0 launched on executor multi with executor id 1674 parsl.dataflow.dflow INFO: Standard output for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/0295b3fd-f711-45b5-8206-376d683d07d9_isrForCrosstalkSources_1031704_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/0295b3fd-f711-45b5-8206-376d683d07d9_isrForCrosstalkSources_1031704_61.stderr parsl.dataflow.dflow DEBUG: Task 1674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1674 has memoization hash 517fa78dacead25303a11a5c9740f9ec parsl.dataflow.memoization INFO: Task 1674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1674 try 0 launched on executor multi with executor id 1675 parsl.dataflow.dflow INFO: Standard output for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/2495545e-202f-4ea6-8cfe-83081777a04a_isrForCrosstalkSources_1031701_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/2495545e-202f-4ea6-8cfe-83081777a04a_isrForCrosstalkSources_1031701_34.stderr parsl.dataflow.dflow DEBUG: Task 1675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1675 has memoization hash d1099cbe0fac616dac3272e5f69450dd parsl.dataflow.memoization INFO: Task 1675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1675 try 0 launched on executor multi with executor id 1676 parsl.dataflow.dflow INFO: Standard output for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/fcb0879a-2fcb-4112-a7d8-fe7963c3acb3_isrForCrosstalkSources_1031712_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/fcb0879a-2fcb-4112-a7d8-fe7963c3acb3_isrForCrosstalkSources_1031712_25.stderr parsl.dataflow.dflow DEBUG: Task 1676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1676 has memoization hash 06be6eabce1ac4547199f0533e8ea2df parsl.dataflow.memoization INFO: Task 1676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1676 try 0 launched on executor multi with executor id 1677 parsl.dataflow.dflow INFO: Standard output for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f58b8b8e-ae58-4f95-b4d4-4ef43211b89e_isrForCrosstalkSources_1031703_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f58b8b8e-ae58-4f95-b4d4-4ef43211b89e_isrForCrosstalkSources_1031703_61.stderr parsl.dataflow.dflow DEBUG: Task 1677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1677 has memoization hash 5032f461ac67ddc8b4c8e80fc5c1b9d4 parsl.dataflow.memoization INFO: Task 1677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1677 try 0 launched on executor multi with executor id 1678 parsl.dataflow.dflow INFO: Standard output for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/acf738d2-7bea-4979-84ee-639e16eda996_isrForCrosstalkSources_1031716_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/acf738d2-7bea-4979-84ee-639e16eda996_isrForCrosstalkSources_1031716_51.stderr parsl.dataflow.dflow DEBUG: Task 1678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1678 has memoization hash 37f42f8b26314f94975f330d9b15f435 parsl.dataflow.memoization INFO: Task 1678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1678 try 0 launched on executor multi with executor id 1679 parsl.dataflow.dflow INFO: Standard output for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/2c60f4c4-5c24-4d28-9850-7cdf6ca36c43_isrForCrosstalkSources_1031705_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/2c60f4c4-5c24-4d28-9850-7cdf6ca36c43_isrForCrosstalkSources_1031705_13.stderr parsl.dataflow.dflow DEBUG: Task 1679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1679 has memoization hash 0dd2f0ca70a9ac471d9da9da12eb50cd parsl.dataflow.memoization INFO: Task 1679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1679 try 0 launched on executor multi with executor id 1680 parsl.dataflow.dflow INFO: Standard output for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/aa2d9a40-565b-4b41-b727-dee692877b5b_isrForCrosstalkSources_1031719_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/aa2d9a40-565b-4b41-b727-dee692877b5b_isrForCrosstalkSources_1031719_53.stderr parsl.dataflow.dflow DEBUG: Task 1680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1680 has memoization hash 24c22f46062a2b8abcfb664205227d67 parsl.dataflow.memoization INFO: Task 1680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1680 try 0 launched on executor multi with executor id 1681 parsl.dataflow.dflow INFO: Standard output for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/0c75f447-19cc-4b69-9eb2-61fe6ee7fa44_isrForCrosstalkSources_1031715_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/0c75f447-19cc-4b69-9eb2-61fe6ee7fa44_isrForCrosstalkSources_1031715_13.stderr parsl.dataflow.dflow DEBUG: Task 1681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1681 has memoization hash e32dce065724b9c9f42386f9e29a9fb7 parsl.dataflow.memoization INFO: Task 1681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1681 try 0 launched on executor multi with executor id 1682 parsl.dataflow.dflow INFO: Standard output for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e22cda9c-a47f-4fda-9c57-b47a4f56b688_isrForCrosstalkSources_1031706_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e22cda9c-a47f-4fda-9c57-b47a4f56b688_isrForCrosstalkSources_1031706_10.stderr parsl.dataflow.dflow DEBUG: Task 1682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1682 has memoization hash 7c57dd6ec6b0a142a7cf36aae1005924 parsl.dataflow.memoization INFO: Task 1682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f94bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1682 try 0 launched on executor multi with executor id 1683 parsl.dataflow.dflow INFO: Standard output for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6e5c6fc2-6932-4a59-b0cf-74b2e3b762af_isrForCrosstalkSources_1031733_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6e5c6fc2-6932-4a59-b0cf-74b2e3b762af_isrForCrosstalkSources_1031733_13.stderr parsl.dataflow.dflow DEBUG: Task 1683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1683 has memoization hash b0cb2ffbaf53ba5cd1492386052a8ff2 parsl.dataflow.memoization INFO: Task 1683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9480e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1683 try 0 launched on executor multi with executor id 1684 parsl.dataflow.dflow INFO: Standard output for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/786f6546-1239-406d-a1e2-95970423520e_isrForCrosstalkSources_1031722_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/786f6546-1239-406d-a1e2-95970423520e_isrForCrosstalkSources_1031722_59.stderr parsl.dataflow.dflow DEBUG: Task 1684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1684 has memoization hash 179b85de61a05a49394043c0cdcf91b4 parsl.dataflow.memoization INFO: Task 1684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1684 try 0 launched on executor multi with executor id 1685 parsl.dataflow.dflow INFO: Standard output for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/4108829d-dec0-4bd3-9e6a-0de7fe6bb6ed_isrForCrosstalkSources_1031701_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/4108829d-dec0-4bd3-9e6a-0de7fe6bb6ed_isrForCrosstalkSources_1031701_45.stderr parsl.dataflow.dflow DEBUG: Task 1685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1685 has memoization hash f9246f2ce9a48a8963a7c4e226cf6dc8 parsl.dataflow.memoization INFO: Task 1685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1685 try 0 launched on executor multi with executor id 1686 parsl.dataflow.dflow INFO: Standard output for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/043b3f37-fb00-4bb5-b712-f5a89212b7ed_isrForCrosstalkSources_1031733_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/043b3f37-fb00-4bb5-b712-f5a89212b7ed_isrForCrosstalkSources_1031733_34.stderr parsl.dataflow.dflow DEBUG: Task 1686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1686 has memoization hash a99ad990634e2eb824a1f832dbd702bd parsl.dataflow.memoization INFO: Task 1686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1686 try 0 launched on executor multi with executor id 1687 parsl.dataflow.dflow INFO: Standard output for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/89621e6c-8956-43ce-93ee-891996465707_isrForCrosstalkSources_1031715_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/89621e6c-8956-43ce-93ee-891996465707_isrForCrosstalkSources_1031715_55.stderr parsl.dataflow.dflow DEBUG: Task 1687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1687 has memoization hash 2651c96e75ee79e8c8ee9b23255c7260 parsl.dataflow.memoization INFO: Task 1687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1687 try 0 launched on executor multi with executor id 1688 parsl.dataflow.dflow INFO: Standard output for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/486ef142-987b-471c-b9af-eedaf34adca7_isrForCrosstalkSources_1031710_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/486ef142-987b-471c-b9af-eedaf34adca7_isrForCrosstalkSources_1031710_34.stderr parsl.dataflow.dflow DEBUG: Task 1688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1688 has memoization hash c90bd9c5286514cd9cf98566db317746 parsl.dataflow.memoization INFO: Task 1688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1688 try 0 launched on executor multi with executor id 1689 parsl.dataflow.dflow INFO: Standard output for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/9943b647-8275-4f2f-b491-1c2c26009bfd_isrForCrosstalkSources_1031732_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/9943b647-8275-4f2f-b491-1c2c26009bfd_isrForCrosstalkSources_1031732_33.stderr parsl.dataflow.dflow DEBUG: Task 1689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1689 has memoization hash aae8d93a6a30f0f102566921d3d6d5d2 parsl.dataflow.memoization INFO: Task 1689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1689 try 0 launched on executor multi with executor id 1690 parsl.dataflow.dflow INFO: Standard output for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7d312f4b-a508-4df8-86ab-01823f3985bf_isrForCrosstalkSources_1031701_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7d312f4b-a508-4df8-86ab-01823f3985bf_isrForCrosstalkSources_1031701_40.stderr parsl.dataflow.dflow DEBUG: Task 1690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1690 has memoization hash 975d4f74570c17161ea9590e905b041f parsl.dataflow.memoization INFO: Task 1690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1690 try 0 launched on executor multi with executor id 1691 parsl.dataflow.dflow INFO: Standard output for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/999cc33d-0315-4c60-879f-b8a4af311278_isrForCrosstalkSources_1031699_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/999cc33d-0315-4c60-879f-b8a4af311278_isrForCrosstalkSources_1031699_25.stderr parsl.dataflow.dflow DEBUG: Task 1691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1691 has memoization hash 600830826af2c55c35f9d3c676bf88ff parsl.dataflow.memoization INFO: Task 1691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1691 try 0 launched on executor multi with executor id 1692 parsl.dataflow.dflow INFO: Standard output for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/208063c6-106e-48ae-b312-914cfd099027_isrForCrosstalkSources_1031732_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/208063c6-106e-48ae-b312-914cfd099027_isrForCrosstalkSources_1031732_49.stderr parsl.dataflow.dflow DEBUG: Task 1692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1692 has memoization hash 805eb0eaa689adebd6eb444a2747a398 parsl.dataflow.memoization INFO: Task 1692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1692 try 0 launched on executor multi with executor id 1693 parsl.dataflow.dflow INFO: Standard output for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/beb39741-be98-41ab-a22a-292b6e43081e_isrForCrosstalkSources_1031700_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/beb39741-be98-41ab-a22a-292b6e43081e_isrForCrosstalkSources_1031700_44.stderr parsl.dataflow.dflow DEBUG: Task 1693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1693 has memoization hash b969fc8791169f8b58c6affb029870f9 parsl.dataflow.memoization INFO: Task 1693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1693 try 0 launched on executor multi with executor id 1694 parsl.dataflow.dflow INFO: Standard output for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/750c86ff-e63c-47cb-a6ba-1bb9d98a1736_isrForCrosstalkSources_1031738_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/750c86ff-e63c-47cb-a6ba-1bb9d98a1736_isrForCrosstalkSources_1031738_13.stderr parsl.dataflow.dflow DEBUG: Task 1694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1694 has memoization hash 9e3ee2cf6e5d37be272492f31eb8399b parsl.dataflow.memoization INFO: Task 1694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1694 try 0 launched on executor multi with executor id 1695 parsl.dataflow.dflow INFO: Standard output for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1508f187-6b74-4126-beec-a669866aac7a_isrForCrosstalkSources_1031700_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1508f187-6b74-4126-beec-a669866aac7a_isrForCrosstalkSources_1031700_11.stderr parsl.dataflow.dflow DEBUG: Task 1695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1695 has memoization hash 56aef0d6ff5ff7a1b773c61e0572d66b parsl.dataflow.memoization INFO: Task 1695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1695 try 0 launched on executor multi with executor id 1696 parsl.dataflow.dflow INFO: Standard output for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d2500e12-7889-4475-bd08-11c1611db1dc_isrForCrosstalkSources_1031738_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d2500e12-7889-4475-bd08-11c1611db1dc_isrForCrosstalkSources_1031738_61.stderr parsl.dataflow.dflow DEBUG: Task 1696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1696 has memoization hash 2664083b53722a087720af4d40a7692b parsl.dataflow.memoization INFO: Task 1696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1696 try 0 launched on executor multi with executor id 1697 parsl.dataflow.dflow INFO: Standard output for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eeeeea7b-8fff-4645-947a-fb1065282358_isrForCrosstalkSources_1031719_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eeeeea7b-8fff-4645-947a-fb1065282358_isrForCrosstalkSources_1031719_51.stderr parsl.dataflow.dflow DEBUG: Task 1697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1697 has memoization hash c8dea76df2d5a0ab9c1a9420f176a035 parsl.dataflow.memoization INFO: Task 1697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1697 try 0 launched on executor multi with executor id 1698 parsl.dataflow.dflow INFO: Standard output for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/5cbe7861-aa9d-4938-8dbd-5aad51ec2be8_isrForCrosstalkSources_1031715_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/5cbe7861-aa9d-4938-8dbd-5aad51ec2be8_isrForCrosstalkSources_1031715_43.stderr parsl.dataflow.dflow DEBUG: Task 1698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1698 has memoization hash 0d980893c8753413598cae422de24b34 parsl.dataflow.memoization INFO: Task 1698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1698 try 0 launched on executor multi with executor id 1699 parsl.dataflow.dflow INFO: Standard output for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a7333962-05d1-4451-baa0-04795cc7930d_isrForCrosstalkSources_1031720_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a7333962-05d1-4451-baa0-04795cc7930d_isrForCrosstalkSources_1031720_9.stderr parsl.dataflow.dflow DEBUG: Task 1699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1699 has memoization hash 65f9d2533a8fbe282bb106150f8747f3 parsl.dataflow.memoization INFO: Task 1699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1699 try 0 launched on executor multi with executor id 1700 parsl.dataflow.dflow INFO: Standard output for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/96c43b4a-8608-47b6-b586-3fc5031bd7ec_isrForCrosstalkSources_1031723_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/96c43b4a-8608-47b6-b586-3fc5031bd7ec_isrForCrosstalkSources_1031723_43.stderr parsl.dataflow.dflow DEBUG: Task 1700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1700 has memoization hash bd1a590d65ccaeb1822586e6d1394b30 parsl.dataflow.memoization INFO: Task 1700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1700 try 0 launched on executor multi with executor id 1701 parsl.dataflow.dflow INFO: Standard output for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fbfe3858-46aa-41e9-9d1b-2efef636a63d_isrForCrosstalkSources_1031700_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fbfe3858-46aa-41e9-9d1b-2efef636a63d_isrForCrosstalkSources_1031700_31.stderr parsl.dataflow.dflow DEBUG: Task 1701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1701 has memoization hash d0c41314369084a267444070efe2b7ee parsl.dataflow.memoization INFO: Task 1701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1701 try 0 launched on executor multi with executor id 1702 parsl.dataflow.dflow INFO: Standard output for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cbc8d656-086c-4bda-b759-350e2099b6c3_isrForCrosstalkSources_1031728_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cbc8d656-086c-4bda-b759-350e2099b6c3_isrForCrosstalkSources_1031728_62.stderr parsl.dataflow.dflow DEBUG: Task 1702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1702 has memoization hash 8c177e24cb94b8ee95ead5c089b28022 parsl.dataflow.memoization INFO: Task 1702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1702 try 0 launched on executor multi with executor id 1703 parsl.dataflow.dflow INFO: Standard output for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4f594657-e528-4095-b685-3c22e031b10b_isrForCrosstalkSources_1031719_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4f594657-e528-4095-b685-3c22e031b10b_isrForCrosstalkSources_1031719_56.stderr parsl.dataflow.dflow DEBUG: Task 1703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1703 has memoization hash 340f96eb11ee4d698e1e6ec307dcf90e parsl.dataflow.memoization INFO: Task 1703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1703 try 0 launched on executor multi with executor id 1704 parsl.dataflow.dflow INFO: Standard output for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/81fd8599-7270-41f1-aacf-9bcc38862623_isrForCrosstalkSources_1031707_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/81fd8599-7270-41f1-aacf-9bcc38862623_isrForCrosstalkSources_1031707_15.stderr parsl.dataflow.dflow DEBUG: Task 1704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1704 has memoization hash fa9006a2f226596781eabf22a072d221 parsl.dataflow.memoization INFO: Task 1704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1704 try 0 launched on executor multi with executor id 1705 parsl.dataflow.dflow INFO: Standard output for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7b946bdc-9a4b-4f8b-8df1-65629936d958_isrForCrosstalkSources_1031729_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7b946bdc-9a4b-4f8b-8df1-65629936d958_isrForCrosstalkSources_1031729_22.stderr parsl.dataflow.dflow DEBUG: Task 1705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1705 has memoization hash 867619e452e416fc1c4d6159793edd47 parsl.dataflow.memoization INFO: Task 1705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1705 try 0 launched on executor multi with executor id 1706 parsl.dataflow.dflow INFO: Standard output for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8903fa01-6d48-44f9-b8c8-e534dd994de7_isrForCrosstalkSources_1031730_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8903fa01-6d48-44f9-b8c8-e534dd994de7_isrForCrosstalkSources_1031730_46.stderr parsl.dataflow.dflow DEBUG: Task 1706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1706 has memoization hash 8b41d4f04202bb9c480138925b9152c8 parsl.dataflow.memoization INFO: Task 1706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1706 try 0 launched on executor multi with executor id 1707 parsl.dataflow.dflow INFO: Standard output for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/95e5005b-f304-4c1d-9185-6141fd4b94c7_isrForCrosstalkSources_1031707_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/95e5005b-f304-4c1d-9185-6141fd4b94c7_isrForCrosstalkSources_1031707_13.stderr parsl.dataflow.dflow DEBUG: Task 1707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1707 has memoization hash 9630ed5001f0d54328e72b754911d952 parsl.dataflow.memoization INFO: Task 1707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1707 try 0 launched on executor multi with executor id 1708 parsl.dataflow.dflow INFO: Standard output for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d18c024a-4176-4d80-836c-7960cda4c9a5_isrForCrosstalkSources_1031731_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d18c024a-4176-4d80-836c-7960cda4c9a5_isrForCrosstalkSources_1031731_3.stderr parsl.dataflow.dflow DEBUG: Task 1708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1708 has memoization hash 947a1da46d4367839e08fea3d30a85b5 parsl.dataflow.memoization INFO: Task 1708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1708 try 0 launched on executor multi with executor id 1709 parsl.dataflow.dflow INFO: Standard output for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/633745d9-b2ea-4d25-ad49-d45fc0c40c99_isrForCrosstalkSources_1031709_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/633745d9-b2ea-4d25-ad49-d45fc0c40c99_isrForCrosstalkSources_1031709_56.stderr parsl.dataflow.dflow DEBUG: Task 1709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1709 has memoization hash f636baf01a94792007a43fceb1dea3ed parsl.dataflow.memoization INFO: Task 1709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:17 proc_lsst.multi:170 [INFO] [multi] job_id 17147953 proc_lsst.multi INFO: [multi] job_id 17147953 2024-03-17 04:02:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-17 04:02:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147953 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147953 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17147953 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-17 04:02:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-17 04:02:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-17 04:02:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1709 try 0 launched on executor multi with executor id 1710 parsl.dataflow.dflow INFO: Standard output for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/fb1ae897-9c10-487c-93dc-e576cbc151be_isrForCrosstalkSources_1031701_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/fb1ae897-9c10-487c-93dc-e576cbc151be_isrForCrosstalkSources_1031701_27.stderr parsl.dataflow.dflow DEBUG: Task 1710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1710 has memoization hash b89645e24ace82aeecd8f1bb7a175ab4 parsl.dataflow.memoization INFO: Task 1710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1710 try 0 launched on executor multi with executor id 1711 parsl.dataflow.dflow INFO: Standard output for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/83d75930-e5f7-4897-b675-a32fcf1bb107_isrForCrosstalkSources_1031706_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/83d75930-e5f7-4897-b675-a32fcf1bb107_isrForCrosstalkSources_1031706_29.stderr parsl.dataflow.dflow DEBUG: Task 1711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1711 has memoization hash 2880cdefe32871cf69cbe24e030636a8 parsl.dataflow.memoization INFO: Task 1711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1711 try 0 launched on executor multi with executor id 1712 parsl.dataflow.dflow INFO: Standard output for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/058250f1-5cbe-4f01-aa79-ec32354f9dd9_isrForCrosstalkSources_1031713_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/058250f1-5cbe-4f01-aa79-ec32354f9dd9_isrForCrosstalkSources_1031713_15.stderr parsl.dataflow.dflow DEBUG: Task 1712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1712 has memoization hash 3a13281daae74d4743a5496e6581e1f6 parsl.dataflow.memoization INFO: Task 1712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1712 try 0 launched on executor multi with executor id 1713 parsl.dataflow.dflow INFO: Standard output for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2021aed1-1aa8-46ea-83cd-3bc8f9695e8b_isrForCrosstalkSources_1031703_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2021aed1-1aa8-46ea-83cd-3bc8f9695e8b_isrForCrosstalkSources_1031703_41.stderr parsl.dataflow.dflow DEBUG: Task 1713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1713 has memoization hash 2baf6bd733af3d3b59d2a631f0aeadc3 parsl.dataflow.memoization INFO: Task 1713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1713 try 0 launched on executor multi with executor id 1714 parsl.dataflow.dflow INFO: Standard output for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9ac3e498-801d-4cf1-b386-e35abaa3ce33_isrForCrosstalkSources_1031729_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9ac3e498-801d-4cf1-b386-e35abaa3ce33_isrForCrosstalkSources_1031729_6.stderr parsl.dataflow.dflow DEBUG: Task 1714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1714 has memoization hash e6daf31c845edead59b786d56be0ad70 parsl.dataflow.memoization INFO: Task 1714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1714 try 0 launched on executor multi with executor id 1715 parsl.dataflow.dflow INFO: Standard output for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f814af3e-4b99-490a-8be0-49ee9bd00d56_isrForCrosstalkSources_1031720_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f814af3e-4b99-490a-8be0-49ee9bd00d56_isrForCrosstalkSources_1031720_4.stderr parsl.dataflow.dflow DEBUG: Task 1715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1715 has memoization hash 013f63338c797ef50e0d6d999b734d93 parsl.dataflow.memoization INFO: Task 1715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9b5fad45d6d5692b7db5a97d21889ed with result from task 0 parsl.dataflow.dflow INFO: Parsl task 1715 try 0 launched on executor multi with executor id 1716 parsl.dataflow.dflow INFO: Standard output for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7c97ff6c-49c7-4440-b559-f9d7079573c3_isrForCrosstalkSources_1031725_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7c97ff6c-49c7-4440-b559-f9d7079573c3_isrForCrosstalkSources_1031725_7.stderr parsl.dataflow.dflow DEBUG: Task 1716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1716 has memoization hash 2b29f5244cd9703664a24c7498065cdf parsl.dataflow.memoization INFO: Task 1716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1716 try 0 launched on executor multi with executor id 1717 parsl.dataflow.dflow INFO: Standard output for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/792170c2-dd73-446f-bc4d-03bd68ed3256_isrForCrosstalkSources_1031713_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/792170c2-dd73-446f-bc4d-03bd68ed3256_isrForCrosstalkSources_1031713_5.stderr parsl.dataflow.dflow DEBUG: Task 1717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1717 has memoization hash a80f04e073d09a3a5db9462d482bb39f parsl.dataflow.memoization INFO: Task 1717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ad75f0e0-51a2-47ab-ad1d-9a8ba608627b_isrForCrosstalkSources_1031726_54.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ad75f0e0-51a2-47ab-ad1d-9a8ba608627b_isrForCrosstalkSources_1031726_54.stderr parsl.dataflow.dflow INFO: Parsl task 1717 try 0 launched on executor multi with executor id 1718 parsl.dataflow.dflow INFO: Standard output for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/c3bcdda6-1c6e-4d8a-b02a-af62b238096d_isrForCrosstalkSources_1031720_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/c3bcdda6-1c6e-4d8a-b02a-af62b238096d_isrForCrosstalkSources_1031720_32.stderr parsl.dataflow.dflow DEBUG: Task 1718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1718 has memoization hash 0726d6563695217bdc011265ae673dbb parsl.dataflow.memoization INFO: Task 1718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1718 try 0 launched on executor multi with executor id 1719 parsl.dataflow.dflow INFO: Standard output for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/6910db0b-a839-4042-9ea8-7b6e7fed13c6_isrForCrosstalkSources_1031703_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/6910db0b-a839-4042-9ea8-7b6e7fed13c6_isrForCrosstalkSources_1031703_35.stderr parsl.dataflow.dflow DEBUG: Task 1719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1719 has memoization hash ab14850294e8c9b8395d11a376bd2604 parsl.dataflow.memoization INFO: Task 1719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1719 try 0 launched on executor multi with executor id 1720 parsl.dataflow.dflow INFO: Standard output for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/55bb91cc-c320-4018-888d-a838b051d51e_isrForCrosstalkSources_1031711_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/55bb91cc-c320-4018-888d-a838b051d51e_isrForCrosstalkSources_1031711_37.stderr parsl.dataflow.dflow DEBUG: Task 1720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1720 has memoization hash b9f6e7f7863062c7e8e3496b75436d78 parsl.dataflow.memoization INFO: Task 1720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1720 try 0 launched on executor multi with executor id 1721 parsl.dataflow.dflow INFO: Standard output for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/ac3be557-4379-4bbf-867a-fa620157b7d3_isrForCrosstalkSources_1031714_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/ac3be557-4379-4bbf-867a-fa620157b7d3_isrForCrosstalkSources_1031714_26.stderr parsl.dataflow.dflow DEBUG: Task 1721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1721 has memoization hash c850cbc5bc7cce5bdcc62c1a298d6a6e parsl.dataflow.memoization INFO: Task 1721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1721 try 0 launched on executor multi with executor id 1722 parsl.dataflow.dflow INFO: Standard output for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a80ba9f7-da70-448a-9053-cae7a836517e_isrForCrosstalkSources_1031736_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a80ba9f7-da70-448a-9053-cae7a836517e_isrForCrosstalkSources_1031736_18.stderr parsl.dataflow.dflow DEBUG: Task 1722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1722 has memoization hash dd607ce36ef3246f51d98768a7edfe9f parsl.dataflow.memoization INFO: Task 1722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1722 try 0 launched on executor multi with executor id 1723 parsl.dataflow.dflow INFO: Standard output for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4968f366-573d-40f4-aa2c-c1e51e3b0e4f_isrForCrosstalkSources_1031738_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4968f366-573d-40f4-aa2c-c1e51e3b0e4f_isrForCrosstalkSources_1031738_49.stderr parsl.dataflow.dflow DEBUG: Task 1723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1723 has memoization hash bd8674ed3c29cd2ceac57c4da54da6f0 parsl.dataflow.memoization INFO: Task 1723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1723 try 0 launched on executor multi with executor id 1724 parsl.dataflow.dflow INFO: Standard output for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/af2c9736-ccb7-4450-9de1-284d20bef359_isrForCrosstalkSources_1031727_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/af2c9736-ccb7-4450-9de1-284d20bef359_isrForCrosstalkSources_1031727_1.stderr parsl.dataflow.dflow DEBUG: Task 1724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1724 has memoization hash bea27a8ce444ce4ccbe45d7b5a0c64b1 parsl.dataflow.memoization INFO: Task 1724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1724 try 0 launched on executor multi with executor id 1725 parsl.dataflow.dflow INFO: Standard output for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/0491953e-3e62-455f-83d3-1c138d15eb0d_isrForCrosstalkSources_1031699_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/0491953e-3e62-455f-83d3-1c138d15eb0d_isrForCrosstalkSources_1031699_3.stderr parsl.dataflow.dflow DEBUG: Task 1725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1725 has memoization hash 1ee57144c484ad4371f976257a0091ac parsl.dataflow.memoization INFO: Task 1725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1725 try 0 launched on executor multi with executor id 1726 parsl.dataflow.dflow INFO: Standard output for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e4890cf5-d366-4a82-84e0-954a927f81c0_isrForCrosstalkSources_1031707_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e4890cf5-d366-4a82-84e0-954a927f81c0_isrForCrosstalkSources_1031707_16.stderr parsl.dataflow.dflow DEBUG: Task 1726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1726 has memoization hash 106ab3c2c14c6a6e5779bdb0aca4c47a parsl.dataflow.memoization INFO: Task 1726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1726 try 0 launched on executor multi with executor id 1727 parsl.dataflow.dflow INFO: Standard output for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/8945d369-5a2c-44d5-af6f-707424a0ef3e_isrForCrosstalkSources_1031711_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/8945d369-5a2c-44d5-af6f-707424a0ef3e_isrForCrosstalkSources_1031711_1.stderr parsl.dataflow.dflow DEBUG: Task 1727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1727 has memoization hash 942b1436b5e95660293f8e0e83e106bf parsl.dataflow.memoization INFO: Task 1727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1727 try 0 launched on executor multi with executor id 1728 parsl.dataflow.dflow INFO: Standard output for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6b750332-8b4d-48a6-8c43-40896a4bd0c2_isrForCrosstalkSources_1031733_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6b750332-8b4d-48a6-8c43-40896a4bd0c2_isrForCrosstalkSources_1031733_24.stderr parsl.dataflow.dflow DEBUG: Task 1728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1728 has memoization hash 1b091b6f7566ebf0045e11649973203d parsl.dataflow.memoization INFO: Task 1728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1728 try 0 launched on executor multi with executor id 1729 parsl.dataflow.dflow INFO: Standard output for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d3c3604b-e0c6-4b49-aa58-05f951031b95_isrForCrosstalkSources_1031725_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d3c3604b-e0c6-4b49-aa58-05f951031b95_isrForCrosstalkSources_1031725_17.stderr parsl.dataflow.dflow DEBUG: Task 1729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1729 has memoization hash ba4241c3d5347dfd10980fc1536e7ce0 parsl.dataflow.memoization INFO: Task 1729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1729 try 0 launched on executor multi with executor id 1730 parsl.dataflow.dflow INFO: Standard output for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/672c5c41-14d5-4308-80dd-5d67093dbdbe_isrForCrosstalkSources_1031718_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/672c5c41-14d5-4308-80dd-5d67093dbdbe_isrForCrosstalkSources_1031718_54.stderr parsl.dataflow.dflow DEBUG: Task 1730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1730 has memoization hash 11a676d7a85786f46f90d377bb438392 parsl.dataflow.memoization INFO: Task 1730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9ddb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1730 try 0 launched on executor multi with executor id 1731 parsl.dataflow.dflow INFO: Standard output for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/a1dd9605-b03e-4d78-b284-e00ace4c0a37_isrForCrosstalkSources_1031727_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/a1dd9605-b03e-4d78-b284-e00ace4c0a37_isrForCrosstalkSources_1031727_34.stderr parsl.dataflow.dflow DEBUG: Task 1731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1731 has memoization hash 8f2e5478ad1b1ea0721e633c9a945e26 parsl.dataflow.memoization INFO: Task 1731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9ddbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1731 try 0 launched on executor multi with executor id 1732 parsl.dataflow.dflow INFO: Standard output for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/933b37e0-50dc-4d44-9737-4ba7de128bdf_isrForCrosstalkSources_1031737_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/933b37e0-50dc-4d44-9737-4ba7de128bdf_isrForCrosstalkSources_1031737_59.stderr parsl.dataflow.dflow DEBUG: Task 1732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1732 has memoization hash c3e493815f7e9593d0c9b43d8ed0e37e parsl.dataflow.memoization INFO: Task 1732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9ddc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1732 try 0 launched on executor multi with executor id 1733 parsl.dataflow.dflow INFO: Standard output for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/b3648bfd-e471-49f9-99ca-d078b684a638_isrForCrosstalkSources_1031730_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/b3648bfd-e471-49f9-99ca-d078b684a638_isrForCrosstalkSources_1031730_43.stderr parsl.dataflow.dflow DEBUG: Task 1733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1733 has memoization hash 29f5260ab844f57ff8fdaa10a476225f parsl.dataflow.memoization INFO: Task 1733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9ddd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1733 try 0 launched on executor multi with executor id 1734 parsl.dataflow.dflow INFO: Standard output for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4bc7d57f-7bfd-4eab-b49c-4e31dfed20c2_isrForCrosstalkSources_1031715_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4bc7d57f-7bfd-4eab-b49c-4e31dfed20c2_isrForCrosstalkSources_1031715_7.stderr parsl.dataflow.dflow DEBUG: Task 1734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1734 has memoization hash 057d1da526febf34b56b01db98eedfd5 parsl.dataflow.memoization INFO: Task 1734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1734 try 0 launched on executor multi with executor id 1735 parsl.dataflow.dflow INFO: Standard output for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/944dab53-6d4a-4abf-bea6-ee6a0606c017_isrForCrosstalkSources_1031706_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/944dab53-6d4a-4abf-bea6-ee6a0606c017_isrForCrosstalkSources_1031706_18.stderr parsl.dataflow.dflow DEBUG: Task 1735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1735 has memoization hash 6fe7620a0868e5d06ad4e13c71f6fe09 parsl.dataflow.memoization INFO: Task 1735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1735 try 0 launched on executor multi with executor id 1736 parsl.dataflow.dflow INFO: Standard output for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/69824f0f-e5b2-47ce-8504-532f571d77ac_isrForCrosstalkSources_1031722_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/69824f0f-e5b2-47ce-8504-532f571d77ac_isrForCrosstalkSources_1031722_1.stderr parsl.dataflow.dflow DEBUG: Task 1736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1736 has memoization hash 5f3c4bcc3cc7126667ddd5eefde548e3 parsl.dataflow.memoization INFO: Task 1736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9ddee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1736 try 0 launched on executor multi with executor id 1737 parsl.dataflow.dflow INFO: Standard output for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/aaa904b3-5763-45dc-92bb-40403cd3a72f_isrForCrosstalkSources_1031717_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/aaa904b3-5763-45dc-92bb-40403cd3a72f_isrForCrosstalkSources_1031717_39.stderr parsl.dataflow.dflow DEBUG: Task 1737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1737 has memoization hash 91281e2a945b3b7fa13ac0e2d4222af4 parsl.dataflow.memoization INFO: Task 1737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9ddf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1737 try 0 launched on executor multi with executor id 1738 parsl.dataflow.dflow INFO: Standard output for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/dc63a10e-8662-45b3-bb37-0650d55cc86b_isrForCrosstalkSources_1031728_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/dc63a10e-8662-45b3-bb37-0650d55cc86b_isrForCrosstalkSources_1031728_52.stderr parsl.dataflow.dflow DEBUG: Task 1738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1738 has memoization hash 07908657d249cd2323726da5bf09f2f3 parsl.dataflow.memoization INFO: Task 1738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1738 try 0 launched on executor multi with executor id 1739 parsl.dataflow.dflow INFO: Standard output for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c714d5ba-e0ef-41b3-a5bc-d224b8f3e872_isrForCrosstalkSources_1031705_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c714d5ba-e0ef-41b3-a5bc-d224b8f3e872_isrForCrosstalkSources_1031705_50.stderr parsl.dataflow.dflow DEBUG: Task 1739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1739 has memoization hash 01a6580e5f0ef7e6976186723f5ff842 parsl.dataflow.memoization INFO: Task 1739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9ddda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1739 try 0 launched on executor multi with executor id 1740 parsl.dataflow.dflow INFO: Standard output for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5341bff8-6a54-48d7-be86-605dbf804f75_isrForCrosstalkSources_1031701_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5341bff8-6a54-48d7-be86-605dbf804f75_isrForCrosstalkSources_1031701_55.stderr parsl.dataflow.dflow DEBUG: Task 1740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1740 has memoization hash 4e732ace353b86a4c3c0a8f2be65b0da parsl.dataflow.memoization INFO: Task 1740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1740 try 0 launched on executor multi with executor id 1741 parsl.dataflow.dflow INFO: Standard output for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/9cb4cfc9-bdc7-4655-95b5-f3b0bc6c6375_isrForCrosstalkSources_1031723_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/9cb4cfc9-bdc7-4655-95b5-f3b0bc6c6375_isrForCrosstalkSources_1031723_18.stderr parsl.dataflow.dflow DEBUG: Task 1741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1741 has memoization hash 356e75378cf6258cd25ac698e919b89a parsl.dataflow.memoization INFO: Task 1741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1741 try 0 launched on executor multi with executor id 1742 parsl.dataflow.dflow INFO: Standard output for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e871bfe2-9bfc-4aa8-9ad0-3376a8bf80cc_isrForCrosstalkSources_1031706_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e871bfe2-9bfc-4aa8-9ad0-3376a8bf80cc_isrForCrosstalkSources_1031706_48.stderr parsl.dataflow.dflow DEBUG: Task 1742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1742 has memoization hash 0673c44fb39227588e0915f267bbe555 parsl.dataflow.memoization INFO: Task 1742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1742 try 0 launched on executor multi with executor id 1743 parsl.dataflow.dflow INFO: Standard output for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/79eb2091-1697-4a50-bc17-e480714583a7_isrForCrosstalkSources_1031704_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/79eb2091-1697-4a50-bc17-e480714583a7_isrForCrosstalkSources_1031704_48.stderr parsl.dataflow.dflow DEBUG: Task 1743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1743 has memoization hash 2f8d088189523ba01cc0d14b29ecfa2d parsl.dataflow.memoization INFO: Task 1743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1743 try 0 launched on executor multi with executor id 1744 parsl.dataflow.dflow INFO: Standard output for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/93290f0f-7645-4b2c-ae8c-a0d3955a3ccf_isrForCrosstalkSources_1031699_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/93290f0f-7645-4b2c-ae8c-a0d3955a3ccf_isrForCrosstalkSources_1031699_30.stderr parsl.dataflow.dflow DEBUG: Task 1744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1744 has memoization hash 714d3e589e7d16e92234cae1afa4f353 parsl.dataflow.memoization INFO: Task 1744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1744 try 0 launched on executor multi with executor id 1745 parsl.dataflow.dflow INFO: Standard output for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/939794d6-4efd-4c10-bad8-ffe1ec56f5b2_isrForCrosstalkSources_1031737_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/939794d6-4efd-4c10-bad8-ffe1ec56f5b2_isrForCrosstalkSources_1031737_22.stderr parsl.dataflow.dflow DEBUG: Task 1745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1745 has memoization hash 2534862fb6448002c653b0cb818eb77a parsl.dataflow.memoization INFO: Task 1745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1745 try 0 launched on executor multi with executor id 1746 parsl.dataflow.dflow INFO: Standard output for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b3695ce4-a64a-44fa-b1db-18abb9cf8e5d_isrForCrosstalkSources_1031712_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b3695ce4-a64a-44fa-b1db-18abb9cf8e5d_isrForCrosstalkSources_1031712_22.stderr parsl.dataflow.dflow DEBUG: Task 1746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1746 has memoization hash 221718351d8007486132a0feb9c20f1d parsl.dataflow.memoization INFO: Task 1746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1746 try 0 launched on executor multi with executor id 1747 parsl.dataflow.dflow INFO: Standard output for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/869c2b7e-bc66-4b80-b13b-15d143cbfa39_isrForCrosstalkSources_1031732_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/869c2b7e-bc66-4b80-b13b-15d143cbfa39_isrForCrosstalkSources_1031732_40.stderr parsl.dataflow.dflow DEBUG: Task 1747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1747 has memoization hash 92a5a15a3e82905e1f0c06cfd44d48d8 parsl.dataflow.memoization INFO: Task 1747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1747 try 0 launched on executor multi with executor id 1748 parsl.dataflow.dflow INFO: Standard output for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c8e4ebab-dd23-497f-b28b-af761c66ea32_isrForCrosstalkSources_1031734_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c8e4ebab-dd23-497f-b28b-af761c66ea32_isrForCrosstalkSources_1031734_33.stderr parsl.dataflow.dflow DEBUG: Task 1748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1748 has memoization hash 31f4d084fc290e85ee175e28e9150c75 parsl.dataflow.memoization INFO: Task 1748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1748 try 0 launched on executor multi with executor id 1749 parsl.dataflow.dflow INFO: Standard output for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/61970c15-0c9a-48de-9ea8-addeb91f0f56_isrForCrosstalkSources_1031728_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/61970c15-0c9a-48de-9ea8-addeb91f0f56_isrForCrosstalkSources_1031728_16.stderr parsl.dataflow.dflow DEBUG: Task 1749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1749 has memoization hash b3a3b1833cdb209a36692a7eeb1af4f1 parsl.dataflow.memoization INFO: Task 1749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1749 try 0 launched on executor multi with executor id 1750 parsl.dataflow.dflow INFO: Standard output for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/58693ca1-4ca8-4a7a-a568-094a3137627f_isrForCrosstalkSources_1031718_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/58693ca1-4ca8-4a7a-a568-094a3137627f_isrForCrosstalkSources_1031718_31.stderr parsl.dataflow.dflow DEBUG: Task 1750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1750 has memoization hash bbc81357af062930d56dc8e5e7efd939 parsl.dataflow.memoization INFO: Task 1750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1750 try 0 launched on executor multi with executor id 1751 parsl.dataflow.dflow INFO: Standard output for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/d80f102f-526d-4f85-994d-6de1c0433a0f_isrForCrosstalkSources_1031732_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/d80f102f-526d-4f85-994d-6de1c0433a0f_isrForCrosstalkSources_1031732_58.stderr parsl.dataflow.dflow DEBUG: Task 1751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1751 has memoization hash b55acc2ead5d19663fcfddf39684250a parsl.dataflow.memoization INFO: Task 1751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1751 try 0 launched on executor multi with executor id 1752 parsl.dataflow.dflow INFO: Standard output for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fddf19bc-302f-40e3-b325-a5b8cc1bef16_isrForCrosstalkSources_1031732_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fddf19bc-302f-40e3-b325-a5b8cc1bef16_isrForCrosstalkSources_1031732_30.stderr parsl.dataflow.dflow DEBUG: Task 1752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1752 has memoization hash 980404b977721198ba74e27cf731eaf6 parsl.dataflow.memoization INFO: Task 1752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1752 try 0 launched on executor multi with executor id 1753 parsl.dataflow.dflow INFO: Standard output for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e6f564c3-223f-40b6-9b36-dacf10141085_isrForCrosstalkSources_1031736_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e6f564c3-223f-40b6-9b36-dacf10141085_isrForCrosstalkSources_1031736_20.stderr parsl.dataflow.dflow DEBUG: Task 1753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1753 has memoization hash 8cf8f24d214201ffbc2507c4901f61a3 parsl.dataflow.memoization INFO: Task 1753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1753 try 0 launched on executor multi with executor id 1754 parsl.dataflow.dflow INFO: Standard output for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/5e549312-5c47-4fe7-8b05-b25a808d7722_isrForCrosstalkSources_1031719_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/5e549312-5c47-4fe7-8b05-b25a808d7722_isrForCrosstalkSources_1031719_57.stderr parsl.dataflow.dflow DEBUG: Task 1754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1754 has memoization hash 5f205e6fede5c8eaa76cb08cf6c727bc parsl.dataflow.memoization INFO: Task 1754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1754 try 0 launched on executor multi with executor id 1755 parsl.dataflow.dflow INFO: Standard output for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c065d253-f737-4a51-9381-3b27c505c3bc_isrForCrosstalkSources_1031732_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c065d253-f737-4a51-9381-3b27c505c3bc_isrForCrosstalkSources_1031732_22.stderr parsl.dataflow.dflow DEBUG: Task 1755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1755 has memoization hash e863365c854a3ea8280b90e027056159 parsl.dataflow.memoization INFO: Task 1755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9deac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1755 try 0 launched on executor multi with executor id 1756 parsl.dataflow.dflow INFO: Standard output for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d04143f1-0eef-4cde-97df-98fb63d6ef52_isrForCrosstalkSources_1031726_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d04143f1-0eef-4cde-97df-98fb63d6ef52_isrForCrosstalkSources_1031726_49.stderr parsl.dataflow.dflow DEBUG: Task 1756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1756 has memoization hash b79a80f136a467649da04af19c0a85f5 parsl.dataflow.memoization INFO: Task 1756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9deb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1756 try 0 launched on executor multi with executor id 1757 parsl.dataflow.dflow INFO: Standard output for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/1c0c48f2-554c-4984-a287-1dab7f2aaa4e_isrForCrosstalkSources_1031734_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/1c0c48f2-554c-4984-a287-1dab7f2aaa4e_isrForCrosstalkSources_1031734_23.stderr parsl.dataflow.dflow DEBUG: Task 1757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1757 has memoization hash 0a69152406c83b7ffac47da9b252b0dc parsl.dataflow.memoization INFO: Task 1757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9de980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1757 try 0 launched on executor multi with executor id 1758 parsl.dataflow.dflow INFO: Standard output for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/e18c0920-1090-4925-a249-d79d3bd5ad1f_isrForCrosstalkSources_1031729_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/e18c0920-1090-4925-a249-d79d3bd5ad1f_isrForCrosstalkSources_1031729_26.stderr parsl.dataflow.dflow DEBUG: Task 1758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1758 has memoization hash dbe92e477e777e8ce35b818595043edc parsl.dataflow.memoization INFO: Task 1758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9ded40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1758 try 0 launched on executor multi with executor id 1759 parsl.dataflow.dflow INFO: Standard output for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/82898554-4c64-4450-a9f6-6892105733df_isrForCrosstalkSources_1031735_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/82898554-4c64-4450-a9f6-6892105733df_isrForCrosstalkSources_1031735_34.stderr parsl.dataflow.dflow DEBUG: Task 1759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1759 has memoization hash 80b59bb47aeb967d54948ab87b568d79 parsl.dataflow.memoization INFO: Task 1759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9deca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1759 try 0 launched on executor multi with executor id 1760 parsl.dataflow.dflow INFO: Standard output for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e6f83fd0-d39b-440d-b5eb-2a96dbee95e4_isrForCrosstalkSources_1031712_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e6f83fd0-d39b-440d-b5eb-2a96dbee95e4_isrForCrosstalkSources_1031712_59.stderr parsl.dataflow.dflow DEBUG: Task 1760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1760 has memoization hash cc63b751ed3a8b5ef64838cd958c94a9 parsl.dataflow.memoization INFO: Task 1760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1760 try 0 launched on executor multi with executor id 1761 parsl.dataflow.dflow INFO: Standard output for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5e19234b-6c5b-4efa-891c-1d44970fbea8_isrForCrosstalkSources_1031728_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5e19234b-6c5b-4efa-891c-1d44970fbea8_isrForCrosstalkSources_1031728_7.stderr parsl.dataflow.dflow DEBUG: Task 1761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1761 has memoization hash 640f9c796a79bf44c93baf0bf762dbca parsl.dataflow.memoization INFO: Task 1761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1761 try 0 launched on executor multi with executor id 1762 parsl.dataflow.dflow INFO: Standard output for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0efa7d15-4300-4f88-8243-fccd9d201a4d_isrForCrosstalkSources_1031706_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0efa7d15-4300-4f88-8243-fccd9d201a4d_isrForCrosstalkSources_1031706_31.stderr parsl.dataflow.dflow DEBUG: Task 1762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1762 has memoization hash 79b34f74d39b385e2eca65b42770e9ad parsl.dataflow.memoization INFO: Task 1762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1762 try 0 launched on executor multi with executor id 1763 parsl.dataflow.dflow INFO: Standard output for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/430ae65c-6f69-48af-9a01-0e004867110c_isrForCrosstalkSources_1031702_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/430ae65c-6f69-48af-9a01-0e004867110c_isrForCrosstalkSources_1031702_37.stderr parsl.dataflow.dflow DEBUG: Task 1763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1763 has memoization hash 77da03198852ff7dc98f836fa32ad11e parsl.dataflow.memoization INFO: Task 1763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1763 try 0 launched on executor multi with executor id 1764 parsl.dataflow.dflow INFO: Standard output for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/13c4179a-4f13-457f-a36e-f0c66033ea98_isrForCrosstalkSources_1031705_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/13c4179a-4f13-457f-a36e-f0c66033ea98_isrForCrosstalkSources_1031705_39.stderr parsl.dataflow.dflow DEBUG: Task 1764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1764 has memoization hash a2d36b378a7c4cf33e5aedee2ed9408e parsl.dataflow.memoization INFO: Task 1764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9def20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1764 try 0 launched on executor multi with executor id 1765 parsl.dataflow.dflow INFO: Standard output for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b8e6f2f4-e085-41c0-a10d-aecad19fc867_isrForCrosstalkSources_1031714_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b8e6f2f4-e085-41c0-a10d-aecad19fc867_isrForCrosstalkSources_1031714_30.stderr parsl.dataflow.dflow DEBUG: Task 1765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1765 has memoization hash 2689b9c663e2f5dd5921807f490324d2 parsl.dataflow.memoization INFO: Task 1765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1765 try 0 launched on executor multi with executor id 1766 parsl.dataflow.dflow INFO: Standard output for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/42f7f87a-e135-48bc-b2d6-69e7f1f5f0a3_isrForCrosstalkSources_1031699_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/42f7f87a-e135-48bc-b2d6-69e7f1f5f0a3_isrForCrosstalkSources_1031699_8.stderr parsl.dataflow.dflow DEBUG: Task 1766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1766 has memoization hash 0fc84ac8fb1c622610450cf903d4fa45 parsl.dataflow.memoization INFO: Task 1766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1766 try 0 launched on executor multi with executor id 1767 parsl.dataflow.dflow INFO: Standard output for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5f7fc1d8-5bd0-4bc9-8dd2-c16bea5d9840_isrForCrosstalkSources_1031724_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5f7fc1d8-5bd0-4bc9-8dd2-c16bea5d9840_isrForCrosstalkSources_1031724_54.stderr parsl.dataflow.dflow DEBUG: Task 1767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1767 has memoization hash 42d4104c947d482f5cf8a49491d4e132 parsl.dataflow.memoization INFO: Task 1767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1767 try 0 launched on executor multi with executor id 1768 parsl.dataflow.dflow INFO: Standard output for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/63f55131-6da6-4d2d-a5b5-4473f5d98d20_isrForCrosstalkSources_1031712_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/63f55131-6da6-4d2d-a5b5-4473f5d98d20_isrForCrosstalkSources_1031712_32.stderr parsl.dataflow.dflow DEBUG: Task 1768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1768 has memoization hash dae868369fc510247c9aff68666d5d58 parsl.dataflow.memoization INFO: Task 1768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1768 try 0 launched on executor multi with executor id 1769 parsl.dataflow.dflow INFO: Standard output for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/b3b955a9-ae25-4f2a-b1a8-a84cd732fd28_isrForCrosstalkSources_1031736_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/b3b955a9-ae25-4f2a-b1a8-a84cd732fd28_isrForCrosstalkSources_1031736_42.stderr parsl.dataflow.dflow DEBUG: Task 1769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1769 has memoization hash a75c868e385b875449c5fe6fdd182f23 parsl.dataflow.memoization INFO: Task 1769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1769 try 0 launched on executor multi with executor id 1770 parsl.dataflow.dflow INFO: Standard output for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/73045bd4-39bd-4fca-8cbd-4c0e5138f83a_isrForCrosstalkSources_1031736_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/73045bd4-39bd-4fca-8cbd-4c0e5138f83a_isrForCrosstalkSources_1031736_12.stderr parsl.dataflow.dflow DEBUG: Task 1770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1770 has memoization hash 96a00502b27a7023f7d423f4fc119d39 parsl.dataflow.memoization INFO: Task 1770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1770 try 0 launched on executor multi with executor id 1771 parsl.dataflow.dflow INFO: Standard output for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/b664984b-c42a-4df6-a5b2-1fa2d11706cb_isrForCrosstalkSources_1031715_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/b664984b-c42a-4df6-a5b2-1fa2d11706cb_isrForCrosstalkSources_1031715_54.stderr parsl.dataflow.dflow DEBUG: Task 1771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1771 has memoization hash 2ae5820cb1e876371eb981e2a9484398 parsl.dataflow.memoization INFO: Task 1771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1771 try 0 launched on executor multi with executor id 1772 parsl.dataflow.dflow INFO: Standard output for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/6c46c321-0d8f-4422-8ea9-0dcc13191584_isrForCrosstalkSources_1031722_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/6c46c321-0d8f-4422-8ea9-0dcc13191584_isrForCrosstalkSources_1031722_18.stderr parsl.dataflow.dflow DEBUG: Task 1772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1772 has memoization hash 81cff3b4a98f00fee29be2f9a9b52f6f parsl.dataflow.memoization INFO: Task 1772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1772 try 0 launched on executor multi with executor id 1773 parsl.dataflow.dflow INFO: Standard output for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/550e8cb9-3738-48dd-8016-4886700bcfb3_isrForCrosstalkSources_1031701_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/550e8cb9-3738-48dd-8016-4886700bcfb3_isrForCrosstalkSources_1031701_3.stderr parsl.dataflow.dflow DEBUG: Task 1773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1773 has memoization hash 4ae185745ba2da3b2dab2c1bf4a71d4d parsl.dataflow.memoization INFO: Task 1773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1773 try 0 launched on executor multi with executor id 1774 parsl.dataflow.dflow INFO: Standard output for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6afc3bff-7c9a-4031-8023-24dfe24ff50f_isrForCrosstalkSources_1031737_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6afc3bff-7c9a-4031-8023-24dfe24ff50f_isrForCrosstalkSources_1031737_45.stderr parsl.dataflow.dflow DEBUG: Task 1774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1774 has memoization hash 6a211da5da476e9fc53d708768e9a0e6 parsl.dataflow.memoization INFO: Task 1774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1774 try 0 launched on executor multi with executor id 1775 parsl.dataflow.dflow INFO: Standard output for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/fd75a39d-dc32-42e8-9334-1592441e1f35_isrForCrosstalkSources_1031735_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/fd75a39d-dc32-42e8-9334-1592441e1f35_isrForCrosstalkSources_1031735_62.stderr parsl.dataflow.dflow DEBUG: Task 1775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1775 has memoization hash 624d2109f6f341db11b90b47c7b5c67b parsl.dataflow.memoization INFO: Task 1775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1775 try 0 launched on executor multi with executor id 1776 parsl.dataflow.dflow INFO: Standard output for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/5d296789-3732-4331-af60-da530473fd21_isrForCrosstalkSources_1031737_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/5d296789-3732-4331-af60-da530473fd21_isrForCrosstalkSources_1031737_25.stderr parsl.dataflow.dflow DEBUG: Task 1776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1776 has memoization hash 52d485ffa9960b4af55084606917e565 parsl.dataflow.memoization INFO: Task 1776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1776 try 0 launched on executor multi with executor id 1777 parsl.dataflow.dflow INFO: Standard output for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/55c59bbd-649d-470c-be23-11d1b483de4b_isrForCrosstalkSources_1031729_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/55c59bbd-649d-470c-be23-11d1b483de4b_isrForCrosstalkSources_1031729_9.stderr parsl.dataflow.dflow DEBUG: Task 1777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1777 has memoization hash b5b5f3d67e065589191061d941ceaeb7 parsl.dataflow.memoization INFO: Task 1777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1777 try 0 launched on executor multi with executor id 1778 parsl.dataflow.dflow INFO: Standard output for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c3864046-2254-486c-b14b-4adae7823330_isrForCrosstalkSources_1031727_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c3864046-2254-486c-b14b-4adae7823330_isrForCrosstalkSources_1031727_4.stderr parsl.dataflow.dflow DEBUG: Task 1778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1778 has memoization hash 13a1ce16fb6c6cbd8bc1a277f41de1f3 parsl.dataflow.memoization INFO: Task 1778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1778 try 0 launched on executor multi with executor id 1779 parsl.dataflow.dflow INFO: Standard output for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/987c701d-ecf9-4cf1-bba4-0a8a6ad84f17_isrForCrosstalkSources_1031736_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/987c701d-ecf9-4cf1-bba4-0a8a6ad84f17_isrForCrosstalkSources_1031736_56.stderr parsl.dataflow.dflow DEBUG: Task 1779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1779 has memoization hash dca2c2d88edf5fb3b1814a583f736a08 parsl.dataflow.memoization INFO: Task 1779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1779 try 0 launched on executor multi with executor id 1780 parsl.dataflow.dflow INFO: Standard output for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/dc992c28-4187-4d8b-bad9-87b8feb90a37_isrForCrosstalkSources_1031699_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/dc992c28-4187-4d8b-bad9-87b8feb90a37_isrForCrosstalkSources_1031699_31.stderr parsl.dataflow.dflow DEBUG: Task 1780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1780 has memoization hash 1c5ca242957f7c9d0495bac83f71623f parsl.dataflow.memoization INFO: Task 1780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1780 try 0 launched on executor multi with executor id 1781 parsl.dataflow.dflow INFO: Standard output for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c7996ae4-3d99-4277-9eb4-9125a0c602df_isrForCrosstalkSources_1031715_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c7996ae4-3d99-4277-9eb4-9125a0c602df_isrForCrosstalkSources_1031715_8.stderr parsl.dataflow.dflow DEBUG: Task 1781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1781 has memoization hash ce2ff8b9f644a42d502a339cc39e0a87 parsl.dataflow.memoization INFO: Task 1781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1781 try 0 launched on executor multi with executor id 1782 parsl.dataflow.dflow INFO: Standard output for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/fc50c121-3635-4a25-8468-ea7b99f22563_isrForCrosstalkSources_1031711_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/fc50c121-3635-4a25-8468-ea7b99f22563_isrForCrosstalkSources_1031711_49.stderr parsl.dataflow.dflow DEBUG: Task 1782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1782 has memoization hash 60ebae850f5bc060175eeda35c80cc0f parsl.dataflow.memoization INFO: Task 1782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1782 try 0 launched on executor multi with executor id 1783 parsl.dataflow.dflow INFO: Standard output for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e306d0c5-355a-4d0e-8857-51cfec600e2d_isrForCrosstalkSources_1031730_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e306d0c5-355a-4d0e-8857-51cfec600e2d_isrForCrosstalkSources_1031730_39.stderr parsl.dataflow.dflow DEBUG: Task 1783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1783 has memoization hash 467b0ce0b78763d501b1d9aaae52e2f9 parsl.dataflow.memoization INFO: Task 1783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1783 try 0 launched on executor multi with executor id 1784 parsl.dataflow.dflow INFO: Standard output for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0574eec4-5dd2-4d45-8a9b-5c7731836cb6_isrForCrosstalkSources_1031705_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0574eec4-5dd2-4d45-8a9b-5c7731836cb6_isrForCrosstalkSources_1031705_25.stderr parsl.dataflow.dflow DEBUG: Task 1784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1784 has memoization hash 41adb23c68eeed9ea9ed95b1f661f8fc parsl.dataflow.memoization INFO: Task 1784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9df1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1784 try 0 launched on executor multi with executor id 1785 parsl.dataflow.dflow INFO: Standard output for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/ad8db44d-42b4-46df-930f-7dca185bdfbc_isrForCrosstalkSources_1031736_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/ad8db44d-42b4-46df-930f-7dca185bdfbc_isrForCrosstalkSources_1031736_31.stderr parsl.dataflow.dflow DEBUG: Task 1785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1785 has memoization hash 2fea4710704072bedf65c4d205dcdd6b parsl.dataflow.memoization INFO: Task 1785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9dd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1785 try 0 launched on executor multi with executor id 1786 parsl.dataflow.dflow INFO: Standard output for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/34b96074-4791-4044-83fd-d29b381834b5_isrForCrosstalkSources_1031705_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/34b96074-4791-4044-83fd-d29b381834b5_isrForCrosstalkSources_1031705_9.stderr parsl.dataflow.dflow DEBUG: Task 1786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1786 has memoization hash 65aee4ecc4bd84ea24bb66f541ef2a11 parsl.dataflow.memoization INFO: Task 1786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8700e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1786 try 0 launched on executor multi with executor id 1787 parsl.dataflow.dflow INFO: Standard output for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/3c0f7323-24ee-4c7c-b815-a858fbcd72f6_isrForCrosstalkSources_1031703_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/3c0f7323-24ee-4c7c-b815-a858fbcd72f6_isrForCrosstalkSources_1031703_26.stderr parsl.dataflow.dflow DEBUG: Task 1787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1787 has memoization hash dfe94c97527d29fd80759e1239660372 parsl.dataflow.memoization INFO: Task 1787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1787 try 0 launched on executor multi with executor id 1788 parsl.dataflow.dflow INFO: Standard output for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d3d3a80c-37eb-4cc3-8d09-b2faebf6509b_isrForCrosstalkSources_1031714_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d3d3a80c-37eb-4cc3-8d09-b2faebf6509b_isrForCrosstalkSources_1031714_28.stderr parsl.dataflow.dflow DEBUG: Task 1788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1788 has memoization hash 8c6a37ca3a47b8ed89758e5d8dbd7498 parsl.dataflow.memoization INFO: Task 1788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1788 try 0 launched on executor multi with executor id 1789 parsl.dataflow.dflow INFO: Standard output for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/feba9287-5917-4627-bff2-5708f3312578_isrForCrosstalkSources_1031702_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/feba9287-5917-4627-bff2-5708f3312578_isrForCrosstalkSources_1031702_48.stderr parsl.dataflow.dflow DEBUG: Task 1789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1789 has memoization hash 5d4446cc431298f7af7865516e6bf985 parsl.dataflow.memoization INFO: Task 1789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8702c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1789 try 0 launched on executor multi with executor id 1790 parsl.dataflow.dflow INFO: Standard output for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/bed12a83-fe40-4c05-87f1-686f89ceb69c_isrForCrosstalkSources_1031718_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/bed12a83-fe40-4c05-87f1-686f89ceb69c_isrForCrosstalkSources_1031718_36.stderr parsl.dataflow.dflow DEBUG: Task 1790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1790 has memoization hash 3ed643d53bc57bf8a6489b2015f6ba9d parsl.dataflow.memoization INFO: Task 1790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1790 try 0 launched on executor multi with executor id 1791 parsl.dataflow.dflow INFO: Standard output for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/565f8398-a2cf-4f68-b6bd-c054dc4c3b9d_isrForCrosstalkSources_1031720_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/565f8398-a2cf-4f68-b6bd-c054dc4c3b9d_isrForCrosstalkSources_1031720_56.stderr parsl.dataflow.dflow DEBUG: Task 1791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1791 has memoization hash 5c2bb3b4385240ee794d63000253ddd4 parsl.dataflow.memoization INFO: Task 1791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1791 try 0 launched on executor multi with executor id 1792 parsl.dataflow.dflow INFO: Standard output for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/76823704-0050-4b0a-a0ae-e17371633e87_isrForCrosstalkSources_1031729_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/76823704-0050-4b0a-a0ae-e17371633e87_isrForCrosstalkSources_1031729_54.stderr parsl.dataflow.dflow DEBUG: Task 1792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1792 has memoization hash b47231b32200b98e767e3f670f182b41 parsl.dataflow.memoization INFO: Task 1792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1792 try 0 launched on executor multi with executor id 1793 parsl.dataflow.dflow INFO: Standard output for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/76fbcedc-9031-4374-ad9e-0d36c9d29b37_isrForCrosstalkSources_1031700_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/76fbcedc-9031-4374-ad9e-0d36c9d29b37_isrForCrosstalkSources_1031700_61.stderr parsl.dataflow.dflow DEBUG: Task 1793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1793 has memoization hash 43e5b57ec6b2908d347144f2a41c00d0 parsl.dataflow.memoization INFO: Task 1793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8704a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1793 try 0 launched on executor multi with executor id 1794 parsl.dataflow.dflow INFO: Standard output for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ccba5059-d92d-4bd6-a230-c37ce6bd5811_isrForCrosstalkSources_1031731_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ccba5059-d92d-4bd6-a230-c37ce6bd5811_isrForCrosstalkSources_1031731_24.stderr parsl.dataflow.dflow DEBUG: Task 1794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1794 has memoization hash c72bc83daab4c6335194d27b52bb584b parsl.dataflow.memoization INFO: Task 1794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8705e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1794 try 0 launched on executor multi with executor id 1795 parsl.dataflow.dflow INFO: Standard output for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7f107a6f-dd7e-4a93-a745-baf209966257_isrForCrosstalkSources_1031725_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7f107a6f-dd7e-4a93-a745-baf209966257_isrForCrosstalkSources_1031725_59.stderr parsl.dataflow.dflow DEBUG: Task 1795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1795 has memoization hash 5fe2aa0763bf6dcf068ade99023e7328 parsl.dataflow.memoization INFO: Task 1795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1795 try 0 launched on executor multi with executor id 1796 parsl.dataflow.dflow INFO: Standard output for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a508ae91-b330-4e6e-a1de-45980d826378_isrForCrosstalkSources_1031712_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a508ae91-b330-4e6e-a1de-45980d826378_isrForCrosstalkSources_1031712_51.stderr parsl.dataflow.dflow DEBUG: Task 1796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1796 has memoization hash b8efb7a76477c522bc6464a9b4459041 parsl.dataflow.memoization INFO: Task 1796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1796 try 0 launched on executor multi with executor id 1797 parsl.dataflow.dflow INFO: Standard output for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b87318ae-0912-45e2-bf54-16b157c547d7_isrForCrosstalkSources_1031718_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b87318ae-0912-45e2-bf54-16b157c547d7_isrForCrosstalkSources_1031718_40.stderr parsl.dataflow.dflow DEBUG: Task 1797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1797 has memoization hash b414f09b61cbe355a92f154644d2dba4 parsl.dataflow.memoization INFO: Task 1797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1797 try 0 launched on executor multi with executor id 1798 parsl.dataflow.dflow INFO: Standard output for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a1cf119a-6f9f-4c89-adc3-5c46ce3732a7_isrForCrosstalkSources_1031722_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a1cf119a-6f9f-4c89-adc3-5c46ce3732a7_isrForCrosstalkSources_1031722_12.stderr parsl.dataflow.dflow DEBUG: Task 1798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1798 has memoization hash 6a0dace2bd07ee770f7723c6370ff211 parsl.dataflow.memoization INFO: Task 1798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8707c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1798 try 0 launched on executor multi with executor id 1799 parsl.dataflow.dflow INFO: Standard output for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6078dcfd-3465-4eb2-b518-38ab70b7e178_isrForCrosstalkSources_1031726_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6078dcfd-3465-4eb2-b518-38ab70b7e178_isrForCrosstalkSources_1031726_32.stderr parsl.dataflow.dflow DEBUG: Task 1799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1799 has memoization hash 911114f21ab024f26ac47442679bec1d parsl.dataflow.memoization INFO: Task 1799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1799 try 0 launched on executor multi with executor id 1800 parsl.dataflow.dflow INFO: Standard output for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ddd5b79a-08fd-4436-98e1-7dae6577d540_isrForCrosstalkSources_1031709_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ddd5b79a-08fd-4436-98e1-7dae6577d540_isrForCrosstalkSources_1031709_36.stderr parsl.dataflow.dflow DEBUG: Task 1800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1800 has memoization hash a85d6c41097ea5f70c0e35260fc02500 parsl.dataflow.memoization INFO: Task 1800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8709a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1800 try 0 launched on executor multi with executor id 1801 parsl.dataflow.dflow INFO: Standard output for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/1065b663-cff9-4af3-b4f4-5944827e5f93_isrForCrosstalkSources_1031704_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/1065b663-cff9-4af3-b4f4-5944827e5f93_isrForCrosstalkSources_1031704_42.stderr parsl.dataflow.dflow DEBUG: Task 1801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1801 has memoization hash 2d282d74c4e83dfa6a7ec30c365609a6 parsl.dataflow.memoization INFO: Task 1801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1801 try 0 launched on executor multi with executor id 1802 parsl.dataflow.dflow INFO: Standard output for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/462fadd5-bab7-4274-aba4-659637430386_isrForCrosstalkSources_1031738_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/462fadd5-bab7-4274-aba4-659637430386_isrForCrosstalkSources_1031738_30.stderr parsl.dataflow.dflow DEBUG: Task 1802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1802 has memoization hash 14ebb24c53c4416d8d2d082ec248e1e7 parsl.dataflow.memoization INFO: Task 1802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1802 try 0 launched on executor multi with executor id 1803 parsl.dataflow.dflow INFO: Standard output for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/92fce5e1-6be4-4735-a4f2-9e1637b33d6b_isrForCrosstalkSources_1031718_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/92fce5e1-6be4-4735-a4f2-9e1637b33d6b_isrForCrosstalkSources_1031718_11.stderr parsl.dataflow.dflow DEBUG: Task 1803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1803 has memoization hash 938b5d6bf3f94b163485e949346b92df parsl.dataflow.memoization INFO: Task 1803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1803 try 0 launched on executor multi with executor id 1804 parsl.dataflow.dflow INFO: Standard output for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f93adf99-fa18-433c-b6fc-f1abe0e9fd1a_isrForCrosstalkSources_1031710_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f93adf99-fa18-433c-b6fc-f1abe0e9fd1a_isrForCrosstalkSources_1031710_58.stderr parsl.dataflow.dflow DEBUG: Task 1804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1804 has memoization hash e913b9250c9d617e7b5c15fbb6abd8b5 parsl.dataflow.memoization INFO: Task 1804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1804 try 0 launched on executor multi with executor id 1805 parsl.dataflow.dflow INFO: Standard output for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9427d0fc-0fa6-42f5-be12-735956fb7f73_isrForCrosstalkSources_1031730_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9427d0fc-0fa6-42f5-be12-735956fb7f73_isrForCrosstalkSources_1031730_45.stderr parsl.dataflow.dflow DEBUG: Task 1805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1805 has memoization hash cea816f6ee285261cbbae7117ef3af50 parsl.dataflow.memoization INFO: Task 1805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1805 try 0 launched on executor multi with executor id 1806 parsl.dataflow.dflow INFO: Standard output for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/147efcba-f90a-4e53-8086-65352cffa4a5_isrForCrosstalkSources_1031712_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/147efcba-f90a-4e53-8086-65352cffa4a5_isrForCrosstalkSources_1031712_50.stderr parsl.dataflow.dflow DEBUG: Task 1806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1806 has memoization hash 56346a5dff703aa167fc07ebc3ae842b parsl.dataflow.memoization INFO: Task 1806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1806 try 0 launched on executor multi with executor id 1807 parsl.dataflow.dflow INFO: Standard output for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/37b22a36-1878-4c02-98f7-c4513454f724_isrForCrosstalkSources_1031720_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/37b22a36-1878-4c02-98f7-c4513454f724_isrForCrosstalkSources_1031720_53.stderr parsl.dataflow.dflow DEBUG: Task 1807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1807 has memoization hash ada7acf3d4fb0005f1b508a5ec510032 parsl.dataflow.memoization INFO: Task 1807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1807 try 0 launched on executor multi with executor id 1808 parsl.dataflow.dflow INFO: Standard output for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/ae73726a-b134-496a-ada7-698c7cf6cbc8_isrForCrosstalkSources_1031710_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/ae73726a-b134-496a-ada7-698c7cf6cbc8_isrForCrosstalkSources_1031710_41.stderr parsl.dataflow.dflow DEBUG: Task 1808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1808 has memoization hash db9f3507b2a51039dc99a134de7e79e2 parsl.dataflow.memoization INFO: Task 1808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1808 try 0 launched on executor multi with executor id 1809 parsl.dataflow.dflow INFO: Standard output for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/20d336fa-37e1-4889-8a49-559ba89c32f6_isrForCrosstalkSources_1031732_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/20d336fa-37e1-4889-8a49-559ba89c32f6_isrForCrosstalkSources_1031732_43.stderr parsl.dataflow.dflow DEBUG: Task 1809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1809 has memoization hash 7bfc86b814f95cc271d0571098845e5f parsl.dataflow.memoization INFO: Task 1809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1809 try 0 launched on executor multi with executor id 1810 parsl.dataflow.dflow INFO: Standard output for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e40041d6-bdcc-484d-9595-acccdc7114aa_isrForCrosstalkSources_1031704_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e40041d6-bdcc-484d-9595-acccdc7114aa_isrForCrosstalkSources_1031704_23.stderr parsl.dataflow.dflow DEBUG: Task 1810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1810 has memoization hash d566d2453125d3af54d94cb4ebd2cb34 parsl.dataflow.memoization INFO: Task 1810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1810 try 0 launched on executor multi with executor id 1811 parsl.dataflow.dflow INFO: Standard output for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/fbd92de9-292e-4aa8-ab2c-d6337dfc21ce_isrForCrosstalkSources_1031711_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/fbd92de9-292e-4aa8-ab2c-d6337dfc21ce_isrForCrosstalkSources_1031711_30.stderr parsl.dataflow.dflow DEBUG: Task 1811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1811 has memoization hash 9ec43b4914323e1324e16a66901638c4 parsl.dataflow.memoization INFO: Task 1811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1811 try 0 launched on executor multi with executor id 1812 parsl.dataflow.dflow INFO: Standard output for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/613043fb-f41a-4488-8894-413fc49fd2fa_isrForCrosstalkSources_1031712_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/613043fb-f41a-4488-8894-413fc49fd2fa_isrForCrosstalkSources_1031712_55.stderr parsl.dataflow.dflow DEBUG: Task 1812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1812 has memoization hash fcf94961d9372337ccb077263d0c3408 parsl.dataflow.memoization INFO: Task 1812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8711c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1812 try 0 launched on executor multi with executor id 1813 parsl.dataflow.dflow INFO: Standard output for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/de22fed1-f3c3-4b3f-82cc-4d82ce123d25_isrForCrosstalkSources_1031717_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/de22fed1-f3c3-4b3f-82cc-4d82ce123d25_isrForCrosstalkSources_1031717_13.stderr parsl.dataflow.dflow DEBUG: Task 1813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1813 has memoization hash f587f4118bd28c2a9d798f65244aad04 parsl.dataflow.memoization INFO: Task 1813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:19 proc_lsst.multi:170 [INFO] [multi] job_id 17147954 proc_lsst.multi INFO: [multi] job_id 17147954 2024-03-17 04:02:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-17 04:02:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147954 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147954 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17147954 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-17 04:02:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-17 04:02:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-17 04:02:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1813 try 0 launched on executor multi with executor id 1814 parsl.dataflow.dflow INFO: Standard output for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/140a1f70-52d1-4f4e-9cb9-c544f4c19ce1_isrForCrosstalkSources_1031712_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/140a1f70-52d1-4f4e-9cb9-c544f4c19ce1_isrForCrosstalkSources_1031712_21.stderr parsl.dataflow.dflow DEBUG: Task 1814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1814 has memoization hash e851bdbe0ff4b5c61620843db4babc06 parsl.dataflow.memoization INFO: Task 1814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fa03ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1814 try 0 launched on executor multi with executor id 1815 parsl.dataflow.dflow INFO: Standard output for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0237b4e9-a4f8-4840-b60b-d1953010b555_isrForCrosstalkSources_1031714_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0237b4e9-a4f8-4840-b60b-d1953010b555_isrForCrosstalkSources_1031714_57.stderr parsl.dataflow.dflow DEBUG: Task 1815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1815 has memoization hash 810d99a9166dae1617f723f914de844e parsl.dataflow.memoization INFO: Task 1815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1815 try 0 launched on executor multi with executor id 1816 parsl.dataflow.dflow INFO: Standard output for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/21c6ee5f-e5c8-49ed-b7dc-becfc78e87e5_isrForCrosstalkSources_1031732_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/21c6ee5f-e5c8-49ed-b7dc-becfc78e87e5_isrForCrosstalkSources_1031732_52.stderr parsl.dataflow.dflow DEBUG: Task 1816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1816 has memoization hash f656544f698b1b52d38ba3219ed6408a parsl.dataflow.memoization INFO: Task 1816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8713a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1816 try 0 launched on executor multi with executor id 1817 parsl.dataflow.dflow INFO: Standard output for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/56f3f81d-d2bc-47cb-9869-81a572c2d8ec_isrForCrosstalkSources_1031718_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/56f3f81d-d2bc-47cb-9869-81a572c2d8ec_isrForCrosstalkSources_1031718_17.stderr parsl.dataflow.dflow DEBUG: Task 1817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1817 has memoization hash 8f90c8fc1861a86d10d9877a28df04c7 parsl.dataflow.memoization INFO: Task 1817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1817 try 0 launched on executor multi with executor id 1818 parsl.dataflow.dflow INFO: Standard output for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/9b6d913d-86bc-4680-95e8-0d517d768334_isrForCrosstalkSources_1031731_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/9b6d913d-86bc-4680-95e8-0d517d768334_isrForCrosstalkSources_1031731_49.stderr parsl.dataflow.dflow DEBUG: Task 1818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1818 has memoization hash 455f571543e098423eac264684b87c09 parsl.dataflow.memoization INFO: Task 1818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1818 try 0 launched on executor multi with executor id 1819 parsl.dataflow.dflow INFO: Standard output for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/1c670202-391e-459f-aac5-50397ec04a5c_isrForCrosstalkSources_1031719_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/1c670202-391e-459f-aac5-50397ec04a5c_isrForCrosstalkSources_1031719_60.stderr parsl.dataflow.dflow DEBUG: Task 1819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1819 has memoization hash a3f93002c67f1be3e365b5417e746350 parsl.dataflow.memoization INFO: Task 1819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1819 try 0 launched on executor multi with executor id 1820 parsl.dataflow.dflow INFO: Standard output for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/346ca8dd-cf68-4a83-99b7-50e30afbe338_isrForCrosstalkSources_1031727_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/346ca8dd-cf68-4a83-99b7-50e30afbe338_isrForCrosstalkSources_1031727_33.stderr parsl.dataflow.dflow DEBUG: Task 1820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1820 has memoization hash 8e1b5c0da2b6b07f2807db38528b3843 parsl.dataflow.memoization INFO: Task 1820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1820 try 0 launched on executor multi with executor id 1821 parsl.dataflow.dflow INFO: Standard output for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/1deefce6-96b2-4fb5-864c-ec53dfb359af_isrForCrosstalkSources_1031710_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/1deefce6-96b2-4fb5-864c-ec53dfb359af_isrForCrosstalkSources_1031710_54.stderr parsl.dataflow.dflow DEBUG: Task 1821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1821 has memoization hash c34e58631165cbfc22c5f594cdf54268 parsl.dataflow.memoization INFO: Task 1821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1821 try 0 launched on executor multi with executor id 1822 parsl.dataflow.dflow INFO: Standard output for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/36fdb4e2-9259-42fa-b052-87c25840ff85_isrForCrosstalkSources_1031724_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/36fdb4e2-9259-42fa-b052-87c25840ff85_isrForCrosstalkSources_1031724_57.stderr parsl.dataflow.dflow DEBUG: Task 1822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1822 has memoization hash c8ccd971d5b029e7d626be4a8755d188 parsl.dataflow.memoization INFO: Task 1822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1822 try 0 launched on executor multi with executor id 1823 parsl.dataflow.dflow INFO: Standard output for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/96418f74-2420-412d-a439-e3fec4088dd7_isrForCrosstalkSources_1031701_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/96418f74-2420-412d-a439-e3fec4088dd7_isrForCrosstalkSources_1031701_28.stderr parsl.dataflow.dflow DEBUG: Task 1823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1823 has memoization hash 6fa7c5dcaf97afa33747f938a1ef1229 parsl.dataflow.memoization INFO: Task 1823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8714e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1823 try 0 launched on executor multi with executor id 1824 parsl.dataflow.dflow INFO: Standard output for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/707d1991-3b44-48bd-9363-8f336d245168_isrForCrosstalkSources_1031717_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/707d1991-3b44-48bd-9363-8f336d245168_isrForCrosstalkSources_1031717_4.stderr parsl.dataflow.dflow DEBUG: Task 1824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1824 has memoization hash 6d0e5c31372c244a0c988f51170c69ec parsl.dataflow.memoization INFO: Task 1824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1824 try 0 launched on executor multi with executor id 1825 parsl.dataflow.dflow INFO: Standard output for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c763f48c-6d15-48fc-a4a4-4bd21eed4f5b_isrForCrosstalkSources_1031731_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c763f48c-6d15-48fc-a4a4-4bd21eed4f5b_isrForCrosstalkSources_1031731_51.stderr parsl.dataflow.dflow DEBUG: Task 1825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1825 has memoization hash 58ff9d36d4233a61855ca22b50a0c7f4 parsl.dataflow.memoization INFO: Task 1825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8716c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1825 try 0 launched on executor multi with executor id 1826 parsl.dataflow.dflow INFO: Standard output for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/1224ea4f-271c-43ab-a17f-65c230d07a0f_isrForCrosstalkSources_1031731_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/1224ea4f-271c-43ab-a17f-65c230d07a0f_isrForCrosstalkSources_1031731_60.stderr parsl.dataflow.dflow DEBUG: Task 1826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1826 has memoization hash 91d506e7598509690c6839fc7d0d345a parsl.dataflow.memoization INFO: Task 1826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1826 try 0 launched on executor multi with executor id 1827 parsl.dataflow.dflow INFO: Standard output for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/5cf090bf-5b37-443f-802a-13e87b78b567_isrForCrosstalkSources_1031708_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/5cf090bf-5b37-443f-802a-13e87b78b567_isrForCrosstalkSources_1031708_33.stderr parsl.dataflow.dflow DEBUG: Task 1827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1827 has memoization hash 67d1cd601a221ee3b4f50069964b48fa parsl.dataflow.memoization INFO: Task 1827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1827 try 0 launched on executor multi with executor id 1828 parsl.dataflow.dflow INFO: Standard output for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/61088146-738e-48f5-8f54-d65681030fb6_isrForCrosstalkSources_1031711_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/61088146-738e-48f5-8f54-d65681030fb6_isrForCrosstalkSources_1031711_47.stderr parsl.dataflow.dflow DEBUG: Task 1828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1828 has memoization hash 6ea1cb1bd34e4d587a45d0e62926379b parsl.dataflow.memoization INFO: Task 1828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1828 try 0 launched on executor multi with executor id 1829 parsl.dataflow.dflow INFO: Standard output for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/192f8f1e-42a6-4dee-9f9c-dfeb71ce4802_isrForCrosstalkSources_1031724_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/192f8f1e-42a6-4dee-9f9c-dfeb71ce4802_isrForCrosstalkSources_1031724_33.stderr parsl.dataflow.dflow DEBUG: Task 1829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1829 has memoization hash 2a0d81d2194c95677b1157f202010a96 parsl.dataflow.memoization INFO: Task 1829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8718a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1829 try 0 launched on executor multi with executor id 1830 parsl.dataflow.dflow INFO: Standard output for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/cacf479f-a9b9-4fcd-9b84-50cddd9b38df_isrForCrosstalkSources_1031724_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/cacf479f-a9b9-4fcd-9b84-50cddd9b38df_isrForCrosstalkSources_1031724_42.stderr parsl.dataflow.dflow DEBUG: Task 1830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1830 has memoization hash 4c958f279700a69e560b2b435a368d32 parsl.dataflow.memoization INFO: Task 1830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1830 try 0 launched on executor multi with executor id 1831 parsl.dataflow.dflow INFO: Standard output for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f9c3a6df-1694-48be-91da-a9aeb1693153_isrForCrosstalkSources_1031730_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f9c3a6df-1694-48be-91da-a9aeb1693153_isrForCrosstalkSources_1031730_5.stderr parsl.dataflow.dflow DEBUG: Task 1831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1831 has memoization hash 5f1e26fd3a7166ad34597d11362923ee parsl.dataflow.memoization INFO: Task 1831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1831 try 0 launched on executor multi with executor id 1832 parsl.dataflow.dflow INFO: Standard output for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/92bf2cbf-5967-40e2-aeb6-fc823486f04c_isrForCrosstalkSources_1031708_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/92bf2cbf-5967-40e2-aeb6-fc823486f04c_isrForCrosstalkSources_1031708_32.stderr parsl.dataflow.dflow DEBUG: Task 1832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1832 has memoization hash e05897ddae28504ad3469e8f3c711e74 parsl.dataflow.memoization INFO: Task 1832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8719e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1832 try 0 launched on executor multi with executor id 1833 parsl.dataflow.dflow INFO: Standard output for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/a27c4a77-9e47-4b00-9169-56267f623f5b_isrForCrosstalkSources_1031708_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/a27c4a77-9e47-4b00-9169-56267f623f5b_isrForCrosstalkSources_1031708_48.stderr parsl.dataflow.dflow DEBUG: Task 1833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1833 has memoization hash 63ffda2009a88f44b780b12064d25e88 parsl.dataflow.memoization INFO: Task 1833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1833 try 0 launched on executor multi with executor id 1834 parsl.dataflow.dflow INFO: Standard output for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0f0ea3f9-5cbf-45e5-bc46-ddb89711a85a_isrForCrosstalkSources_1031719_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0f0ea3f9-5cbf-45e5-bc46-ddb89711a85a_isrForCrosstalkSources_1031719_6.stderr parsl.dataflow.dflow DEBUG: Task 1834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1834 has memoization hash 296965e77486c44f5b4bc3ff2aae90b8 parsl.dataflow.memoization INFO: Task 1834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1834 try 0 launched on executor multi with executor id 1835 parsl.dataflow.dflow INFO: Standard output for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/6a5edb42-7861-42f1-b20c-e611ab73dfe5_isrForCrosstalkSources_1031730_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/6a5edb42-7861-42f1-b20c-e611ab73dfe5_isrForCrosstalkSources_1031730_50.stderr parsl.dataflow.dflow DEBUG: Task 1835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1835 has memoization hash 6bf4299c0274d27ca324a5bb1650686b parsl.dataflow.memoization INFO: Task 1835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1835 try 0 launched on executor multi with executor id 1836 parsl.dataflow.dflow INFO: Standard output for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/fa09d9cc-828c-41f8-899b-ce4bbe00278f_isrForCrosstalkSources_1031709_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/fa09d9cc-828c-41f8-899b-ce4bbe00278f_isrForCrosstalkSources_1031709_1.stderr parsl.dataflow.dflow DEBUG: Task 1836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1836 has memoization hash 41ebe45ddc82306e36c76f1da6fd0f76 parsl.dataflow.memoization INFO: Task 1836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1836 try 0 launched on executor multi with executor id 1837 parsl.dataflow.dflow INFO: Standard output for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1972e91d-0c3a-4de5-95f9-3b5f36be4864_isrForCrosstalkSources_1031703_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1972e91d-0c3a-4de5-95f9-3b5f36be4864_isrForCrosstalkSources_1031703_1.stderr parsl.dataflow.dflow DEBUG: Task 1837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1837 has memoization hash db2e6b9e271a7d2e8103f9ce2f445822 parsl.dataflow.memoization INFO: Task 1837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1837 try 0 launched on executor multi with executor id 1838 parsl.dataflow.dflow INFO: Standard output for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/0e17f5d8-c0a7-471a-8830-b8fc2f851ec3_isrForCrosstalkSources_1031730_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/0e17f5d8-c0a7-471a-8830-b8fc2f851ec3_isrForCrosstalkSources_1031730_53.stderr parsl.dataflow.dflow DEBUG: Task 1838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1838 has memoization hash 78e268025a33c15ce1e365d27f164a29 parsl.dataflow.memoization INFO: Task 1838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1838 try 0 launched on executor multi with executor id 1839 parsl.dataflow.dflow INFO: Standard output for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/9f66dc9c-3aca-49a6-938d-74ad128dea42_isrForCrosstalkSources_1031699_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/9f66dc9c-3aca-49a6-938d-74ad128dea42_isrForCrosstalkSources_1031699_16.stderr parsl.dataflow.dflow DEBUG: Task 1839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1839 has memoization hash 630cf228b5595cdda751f8b5aa8c05ce parsl.dataflow.memoization INFO: Task 1839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1839 try 0 launched on executor multi with executor id 1840 parsl.dataflow.dflow INFO: Standard output for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/56bf2726-41bf-4853-99c4-59bf5ff734ec_isrForCrosstalkSources_1031705_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/56bf2726-41bf-4853-99c4-59bf5ff734ec_isrForCrosstalkSources_1031705_32.stderr parsl.dataflow.dflow DEBUG: Task 1840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1840 has memoization hash 77c78a3440ccd289c1aa9cc30e653998 parsl.dataflow.memoization INFO: Task 1840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1840 try 0 launched on executor multi with executor id 1841 parsl.dataflow.dflow INFO: Standard output for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e48e452b-3512-47c1-9d2f-1d9a8632995c_isrForCrosstalkSources_1031733_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e48e452b-3512-47c1-9d2f-1d9a8632995c_isrForCrosstalkSources_1031733_27.stderr parsl.dataflow.dflow DEBUG: Task 1841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1841 has memoization hash c5e0c7f8be3cf022eb61e446a785dbcb parsl.dataflow.memoization INFO: Task 1841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1841 try 0 launched on executor multi with executor id 1842 parsl.dataflow.dflow INFO: Standard output for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/c69c0137-9716-49b8-b22b-b9f9a2c4e9ae_isrForCrosstalkSources_1031736_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/c69c0137-9716-49b8-b22b-b9f9a2c4e9ae_isrForCrosstalkSources_1031736_57.stderr parsl.dataflow.dflow DEBUG: Task 1842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1842 has memoization hash 9ae9486d0ee019234e89e9b7da469175 parsl.dataflow.memoization INFO: Task 1842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1842 try 0 launched on executor multi with executor id 1843 parsl.dataflow.dflow INFO: Standard output for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/92f9616f-edfc-4793-a1be-840a089e9ae8_isrForCrosstalkSources_1031703_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/92f9616f-edfc-4793-a1be-840a089e9ae8_isrForCrosstalkSources_1031703_18.stderr parsl.dataflow.dflow DEBUG: Task 1843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1843 has memoization hash 4006945da1d73c025167dd32cfdc92ad parsl.dataflow.memoization INFO: Task 1843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1843 try 0 launched on executor multi with executor id 1844 parsl.dataflow.dflow INFO: Standard output for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a8f938b5-7300-427d-97a2-77ef86470395_isrForCrosstalkSources_1031706_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a8f938b5-7300-427d-97a2-77ef86470395_isrForCrosstalkSources_1031706_45.stderr parsl.dataflow.dflow DEBUG: Task 1844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1844 has memoization hash 8e79643442d18f8e14dacf339181ddd7 parsl.dataflow.memoization INFO: Task 1844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8722a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1844 try 0 launched on executor multi with executor id 1845 parsl.dataflow.dflow INFO: Standard output for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f88a9278-3a8f-4609-a3d6-a21883de6726_isrForCrosstalkSources_1031735_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f88a9278-3a8f-4609-a3d6-a21883de6726_isrForCrosstalkSources_1031735_5.stderr parsl.dataflow.dflow DEBUG: Task 1845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1845 has memoization hash 3fe9c60560766b62ba661e40c681489b parsl.dataflow.memoization INFO: Task 1845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8720c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1845 try 0 launched on executor multi with executor id 1846 parsl.dataflow.dflow INFO: Standard output for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/02ed4ea2-f437-4269-8e9d-eba833f9e469_isrForCrosstalkSources_1031719_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/02ed4ea2-f437-4269-8e9d-eba833f9e469_isrForCrosstalkSources_1031719_37.stderr parsl.dataflow.dflow DEBUG: Task 1846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1846 has memoization hash 35d419161389d924281bfc0784767264 parsl.dataflow.memoization INFO: Task 1846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8723e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1846 try 0 launched on executor multi with executor id 1847 parsl.dataflow.dflow INFO: Standard output for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/dfdc2043-c46d-41cf-a227-480cfd354901_isrForCrosstalkSources_1031700_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/dfdc2043-c46d-41cf-a227-480cfd354901_isrForCrosstalkSources_1031700_25.stderr parsl.dataflow.dflow DEBUG: Task 1847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1847 has memoization hash 2cadb511e9a3769b9dbf6de6c4a19fab parsl.dataflow.memoization INFO: Task 1847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1847 try 0 launched on executor multi with executor id 1848 parsl.dataflow.dflow INFO: Standard output for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/eeb41e65-fc85-4d47-8a1b-b5ab96cfed45_isrForCrosstalkSources_1031722_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/eeb41e65-fc85-4d47-8a1b-b5ab96cfed45_isrForCrosstalkSources_1031722_45.stderr parsl.dataflow.dflow DEBUG: Task 1848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1848 has memoization hash d42a6d1bdf6fba61dd5ffe6f63df4c28 parsl.dataflow.memoization INFO: Task 1848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1848 try 0 launched on executor multi with executor id 1849 parsl.dataflow.dflow INFO: Standard output for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/29ebfbc7-4b8d-4df8-a63a-e3ef67bc87f3_isrForCrosstalkSources_1031712_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/29ebfbc7-4b8d-4df8-a63a-e3ef67bc87f3_isrForCrosstalkSources_1031712_10.stderr parsl.dataflow.dflow DEBUG: Task 1849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1849 has memoization hash 974f24606237aa7d4a052aefbdfdc4b4 parsl.dataflow.memoization INFO: Task 1849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8725c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1849 try 0 launched on executor multi with executor id 1850 parsl.dataflow.dflow INFO: Standard output for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/cd2405a2-4103-4473-8645-9221761cb3fb_isrForCrosstalkSources_1031732_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/cd2405a2-4103-4473-8645-9221761cb3fb_isrForCrosstalkSources_1031732_51.stderr parsl.dataflow.dflow DEBUG: Task 1850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1850 has memoization hash ec30492254c8a639f33d0bf932e65326 parsl.dataflow.memoization INFO: Task 1850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1850 try 0 launched on executor multi with executor id 1851 parsl.dataflow.dflow INFO: Standard output for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/628d31f4-d66d-4f43-863a-0297c7315d97_isrForCrosstalkSources_1031722_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/628d31f4-d66d-4f43-863a-0297c7315d97_isrForCrosstalkSources_1031722_24.stderr parsl.dataflow.dflow DEBUG: Task 1851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1851 has memoization hash 7d49ed8335ab680faa5b6905559c0920 parsl.dataflow.memoization INFO: Task 1851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1851 try 0 launched on executor multi with executor id 1852 parsl.dataflow.dflow INFO: Standard output for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/e0f0118d-2fcd-4703-acfc-9335aacd9f79_isrForCrosstalkSources_1031726_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/e0f0118d-2fcd-4703-acfc-9335aacd9f79_isrForCrosstalkSources_1031726_46.stderr parsl.dataflow.dflow DEBUG: Task 1852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1852 has memoization hash 53eaf6102181195f4b19e009a3616fe7 parsl.dataflow.memoization INFO: Task 1852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8727a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1852 try 0 launched on executor multi with executor id 1853 parsl.dataflow.dflow INFO: Standard output for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/359089e1-e7cc-4ed6-818b-632cb00c0113_isrForCrosstalkSources_1031709_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/359089e1-e7cc-4ed6-818b-632cb00c0113_isrForCrosstalkSources_1031709_15.stderr parsl.dataflow.dflow DEBUG: Task 1853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1853 has memoization hash f37decbaced2b38431600ce88ed3c82a parsl.dataflow.memoization INFO: Task 1853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1853 try 0 launched on executor multi with executor id 1854 parsl.dataflow.dflow INFO: Standard output for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7ced5873-6df6-48d8-a380-d1d9231ad0f1_isrForCrosstalkSources_1031710_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7ced5873-6df6-48d8-a380-d1d9231ad0f1_isrForCrosstalkSources_1031710_28.stderr parsl.dataflow.dflow DEBUG: Task 1854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1854 has memoization hash 6c494522d487ab18d7365fe0705f8afc parsl.dataflow.memoization INFO: Task 1854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8728e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1854 try 0 launched on executor multi with executor id 1855 parsl.dataflow.dflow INFO: Standard output for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/bcf4b9f0-28e6-46d0-bd47-8bab74382121_isrForCrosstalkSources_1031720_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/bcf4b9f0-28e6-46d0-bd47-8bab74382121_isrForCrosstalkSources_1031720_27.stderr parsl.dataflow.dflow DEBUG: Task 1855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1855 has memoization hash b34cd289f1b86aed48472b52a3a2ff2e parsl.dataflow.memoization INFO: Task 1855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1855 try 0 launched on executor multi with executor id 1856 parsl.dataflow.dflow INFO: Standard output for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1785112d-d1db-47ad-86a8-012a7b7a3a68_isrForCrosstalkSources_1031713_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1785112d-d1db-47ad-86a8-012a7b7a3a68_isrForCrosstalkSources_1031713_26.stderr parsl.dataflow.dflow DEBUG: Task 1856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1856 has memoization hash 17f3dc376cdfa8e7c835ac8ccc0032da parsl.dataflow.memoization INFO: Task 1856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1856 try 0 launched on executor multi with executor id 1857 parsl.dataflow.dflow INFO: Standard output for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/3eabd279-ea50-4846-872f-3dba0a09b59d_isrForCrosstalkSources_1031722_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/3eabd279-ea50-4846-872f-3dba0a09b59d_isrForCrosstalkSources_1031722_27.stderr parsl.dataflow.dflow DEBUG: Task 1857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1857 has memoization hash b9ee957ee1a0414ab726b77dd0915664 parsl.dataflow.memoization INFO: Task 1857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1857 try 0 launched on executor multi with executor id 1858 parsl.dataflow.dflow INFO: Standard output for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a21d9596-742d-41fb-b93d-ab4df4cf67de_isrForCrosstalkSources_1031723_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a21d9596-742d-41fb-b93d-ab4df4cf67de_isrForCrosstalkSources_1031723_2.stderr parsl.dataflow.dflow DEBUG: Task 1858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1858 has memoization hash 5a533626681d6582a9efaea9b288053c parsl.dataflow.memoization INFO: Task 1858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1858 try 0 launched on executor multi with executor id 1859 parsl.dataflow.dflow INFO: Standard output for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/0029f769-3b8a-44a8-a57c-861d834ff2a7_isrForCrosstalkSources_1031708_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/0029f769-3b8a-44a8-a57c-861d834ff2a7_isrForCrosstalkSources_1031708_61.stderr parsl.dataflow.dflow DEBUG: Task 1859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1859 has memoization hash d385a38240fe28d6e25dfd467127f753 parsl.dataflow.memoization INFO: Task 1859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1859 try 0 launched on executor multi with executor id 1860 parsl.dataflow.dflow INFO: Standard output for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ca347d49-567a-4be6-b6c4-3f51c933388a_isrForCrosstalkSources_1031738_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ca347d49-567a-4be6-b6c4-3f51c933388a_isrForCrosstalkSources_1031738_34.stderr parsl.dataflow.dflow DEBUG: Task 1860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1860 has memoization hash d3d9c0bce6c6eacdbef266e2247a4489 parsl.dataflow.memoization INFO: Task 1860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1860 try 0 launched on executor multi with executor id 1861 parsl.dataflow.dflow INFO: Standard output for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a74db3b1-23bf-4dea-8b12-41a3a2a94877_isrForCrosstalkSources_1031702_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a74db3b1-23bf-4dea-8b12-41a3a2a94877_isrForCrosstalkSources_1031702_49.stderr parsl.dataflow.dflow DEBUG: Task 1861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1861 has memoization hash a700450cc5440055442240689ca09872 parsl.dataflow.memoization INFO: Task 1861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1861 try 0 launched on executor multi with executor id 1862 parsl.dataflow.dflow INFO: Standard output for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c2603019-ec6d-469d-b4a9-fd3a5e3162dd_isrForCrosstalkSources_1031716_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c2603019-ec6d-469d-b4a9-fd3a5e3162dd_isrForCrosstalkSources_1031716_41.stderr parsl.dataflow.dflow DEBUG: Task 1862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1862 has memoization hash 2e1ca4d30a61f4e79f488f435f7cc0bd parsl.dataflow.memoization INFO: Task 1862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1862 try 0 launched on executor multi with executor id 1863 parsl.dataflow.dflow INFO: Standard output for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/51b3efd0-ab85-4662-8c1a-08c6df8104dc_isrForCrosstalkSources_1031729_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/51b3efd0-ab85-4662-8c1a-08c6df8104dc_isrForCrosstalkSources_1031729_38.stderr parsl.dataflow.dflow DEBUG: Task 1863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1863 has memoization hash aa282c5918d4ae0aeac3796bc5d82b9d parsl.dataflow.memoization INFO: Task 1863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1863 try 0 launched on executor multi with executor id 1864 parsl.dataflow.dflow INFO: Standard output for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bf35a201-ea7c-4343-a803-f831dfdabdba_isrForCrosstalkSources_1031723_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bf35a201-ea7c-4343-a803-f831dfdabdba_isrForCrosstalkSources_1031723_44.stderr parsl.dataflow.dflow DEBUG: Task 1864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1864 has memoization hash d9213f298d335ec22171cb3da10e1f12 parsl.dataflow.memoization INFO: Task 1864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1864 try 0 launched on executor multi with executor id 1865 parsl.dataflow.dflow INFO: Standard output for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/44a8f164-fb83-4a4c-b93d-311b74d24597_isrForCrosstalkSources_1031710_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/44a8f164-fb83-4a4c-b93d-311b74d24597_isrForCrosstalkSources_1031710_16.stderr parsl.dataflow.dflow DEBUG: Task 1865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1865 has memoization hash c88c05dea31144edf556608036805e8a parsl.dataflow.memoization INFO: Task 1865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1865 try 0 launched on executor multi with executor id 1866 parsl.dataflow.dflow INFO: Standard output for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2c1305e5-83e9-4cd6-a6e1-1dca25675155_isrForCrosstalkSources_1031734_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2c1305e5-83e9-4cd6-a6e1-1dca25675155_isrForCrosstalkSources_1031734_12.stderr parsl.dataflow.dflow DEBUG: Task 1866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1866 has memoization hash c9aac37aca0efb659ea25ead156e3d1e parsl.dataflow.memoization INFO: Task 1866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1866 try 0 launched on executor multi with executor id 1867 parsl.dataflow.dflow INFO: Standard output for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c0349430-ebe0-4d20-8878-bf038ff66ee2_isrForCrosstalkSources_1031712_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c0349430-ebe0-4d20-8878-bf038ff66ee2_isrForCrosstalkSources_1031712_18.stderr parsl.dataflow.dflow DEBUG: Task 1867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1867 has memoization hash 2a356a8694f4a91b42c8f10af76d5219 parsl.dataflow.memoization INFO: Task 1867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f872f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1867 try 0 launched on executor multi with executor id 1868 parsl.dataflow.dflow INFO: Standard output for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b87595cd-f744-4c36-b524-447ca7109635_isrForCrosstalkSources_1031714_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b87595cd-f744-4c36-b524-447ca7109635_isrForCrosstalkSources_1031714_29.stderr parsl.dataflow.dflow DEBUG: Task 1868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1868 has memoization hash f7d4e7033ff26b1b80f551447507f7dc parsl.dataflow.memoization INFO: Task 1868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1868 try 0 launched on executor multi with executor id 1869 parsl.dataflow.dflow INFO: Standard output for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4e3f2c43-000a-4ecc-9d40-49dc67f121a1_isrForCrosstalkSources_1031717_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4e3f2c43-000a-4ecc-9d40-49dc67f121a1_isrForCrosstalkSources_1031717_50.stderr parsl.dataflow.dflow DEBUG: Task 1869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1869 has memoization hash e6189dcc0c56b29e898522141b75fd2c parsl.dataflow.memoization INFO: Task 1869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8731a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1869 try 0 launched on executor multi with executor id 1870 parsl.dataflow.dflow INFO: Standard output for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/8f26b0bb-06e5-4c49-ac18-128d46ae55a9_isrForCrosstalkSources_1031723_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/8f26b0bb-06e5-4c49-ac18-128d46ae55a9_isrForCrosstalkSources_1031723_7.stderr parsl.dataflow.dflow DEBUG: Task 1870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1870 has memoization hash 3f67ed1ca02d8edf3f1bed6a244ce929 parsl.dataflow.memoization INFO: Task 1870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1870 try 0 launched on executor multi with executor id 1871 parsl.dataflow.dflow INFO: Standard output for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e9843898-5d9c-45b5-83bc-3ec76150ca32_isrForCrosstalkSources_1031736_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e9843898-5d9c-45b5-83bc-3ec76150ca32_isrForCrosstalkSources_1031736_52.stderr parsl.dataflow.dflow DEBUG: Task 1871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1871 has memoization hash a569db072f39d133d14f21bbfdcf37ac parsl.dataflow.memoization INFO: Task 1871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1871 try 0 launched on executor multi with executor id 1872 parsl.dataflow.dflow INFO: Standard output for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e22a1199-38a9-4bb7-b50c-87e98d109a2b_isrForCrosstalkSources_1031706_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e22a1199-38a9-4bb7-b50c-87e98d109a2b_isrForCrosstalkSources_1031706_26.stderr parsl.dataflow.dflow DEBUG: Task 1872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1872 has memoization hash 09e9a08d9ee4171af2c275f093c43036 parsl.dataflow.memoization INFO: Task 1872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1872 try 0 launched on executor multi with executor id 1873 parsl.dataflow.dflow INFO: Standard output for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/e087b8e0-ddc5-4934-a8a9-4b60f1edee55_isrForCrosstalkSources_1031708_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/e087b8e0-ddc5-4934-a8a9-4b60f1edee55_isrForCrosstalkSources_1031708_44.stderr parsl.dataflow.dflow DEBUG: Task 1873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1873 has memoization hash e8cf8abedc90306dbbc6d06699997bba parsl.dataflow.memoization INFO: Task 1873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1873 try 0 launched on executor multi with executor id 1874 parsl.dataflow.dflow INFO: Standard output for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/24f4395b-df3b-4117-834c-256212c8ef6d_isrForCrosstalkSources_1031725_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/24f4395b-df3b-4117-834c-256212c8ef6d_isrForCrosstalkSources_1031725_2.stderr parsl.dataflow.dflow DEBUG: Task 1874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1874 has memoization hash a9083942b5c300a379066a3a12f0a57f parsl.dataflow.memoization INFO: Task 1874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1874 try 0 launched on executor multi with executor id 1875 parsl.dataflow.dflow INFO: Standard output for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/4c5e4d66-7bfc-4904-9de8-913ba22ccfff_isrForCrosstalkSources_1031707_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/4c5e4d66-7bfc-4904-9de8-913ba22ccfff_isrForCrosstalkSources_1031707_21.stderr parsl.dataflow.dflow DEBUG: Task 1875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1875 has memoization hash 2ef2277cf001f2ff36cad9096d5f2077 parsl.dataflow.memoization INFO: Task 1875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1875 try 0 launched on executor multi with executor id 1876 parsl.dataflow.dflow INFO: Standard output for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7e59fdf7-1511-49e0-8cad-7b2eaee3e915_isrForCrosstalkSources_1031710_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7e59fdf7-1511-49e0-8cad-7b2eaee3e915_isrForCrosstalkSources_1031710_42.stderr parsl.dataflow.dflow DEBUG: Task 1876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1876 has memoization hash a2f0e1bcc058fa39e4cb66b51120b0cb parsl.dataflow.memoization INFO: Task 1876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8739c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6df087bd5cccae0cbd467db4548fee2f with result from task 3 parsl.dataflow.dflow INFO: Parsl task 1876 try 0 launched on executor multi with executor id 1877 parsl.dataflow.dflow INFO: Standard output for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/34547987-5674-4473-8848-ab4459a9748f_isrForCrosstalkSources_1031704_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/34547987-5674-4473-8848-ab4459a9748f_isrForCrosstalkSources_1031704_59.stderr parsl.dataflow.dflow DEBUG: Task 1877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1877 has memoization hash 99abb1d08fa82b15a0c5252e3aaf250a parsl.dataflow.memoization INFO: Task 1877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/5a511f79-4777-4c80-8873-bf1909945bfb_isrForCrosstalkSources_1031726_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/5a511f79-4777-4c80-8873-bf1909945bfb_isrForCrosstalkSources_1031726_59.stderr parsl.dataflow.dflow INFO: Parsl task 1877 try 0 launched on executor multi with executor id 1878 parsl.dataflow.dflow INFO: Standard output for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b7e58c0c-3440-4806-8757-c0a29ad377d8_isrForCrosstalkSources_1031728_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b7e58c0c-3440-4806-8757-c0a29ad377d8_isrForCrosstalkSources_1031728_23.stderr parsl.dataflow.dflow DEBUG: Task 1878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1878 has memoization hash 82640de245fe7619a27e547b46073559 parsl.dataflow.memoization INFO: Task 1878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1878 try 0 launched on executor multi with executor id 1879 parsl.dataflow.dflow INFO: Standard output for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/662096ea-743a-4841-a681-72d25ba37612_isrForCrosstalkSources_1031732_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/662096ea-743a-4841-a681-72d25ba37612_isrForCrosstalkSources_1031732_36.stderr parsl.dataflow.dflow DEBUG: Task 1879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1879 has memoization hash 56980c6bc0f5717e240487a10790d951 parsl.dataflow.memoization INFO: Task 1879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1879 try 0 launched on executor multi with executor id 1880 parsl.dataflow.dflow INFO: Standard output for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c2aeadd2-ccf2-4b79-aeac-2f16d3a8dc3e_isrForCrosstalkSources_1031731_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c2aeadd2-ccf2-4b79-aeac-2f16d3a8dc3e_isrForCrosstalkSources_1031731_58.stderr parsl.dataflow.dflow DEBUG: Task 1880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1880 has memoization hash 476cccf4e6e22aeb16ab3017671311b9 parsl.dataflow.memoization INFO: Task 1880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8736a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1880 try 0 launched on executor multi with executor id 1881 parsl.dataflow.dflow INFO: Standard output for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/1658c3c6-4148-4edf-b9c0-75aa157ba141_isrForCrosstalkSources_1031724_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/1658c3c6-4148-4edf-b9c0-75aa157ba141_isrForCrosstalkSources_1031724_24.stderr parsl.dataflow.dflow DEBUG: Task 1881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1881 has memoization hash 3c2211cbbbd9e92ce5bc8e7d2731b0fd parsl.dataflow.memoization INFO: Task 1881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8737e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1881 try 0 launched on executor multi with executor id 1882 parsl.dataflow.dflow INFO: Standard output for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4c9ebb34-a42a-4d81-89fd-c013ab218c26_isrForCrosstalkSources_1031737_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4c9ebb34-a42a-4d81-89fd-c013ab218c26_isrForCrosstalkSources_1031737_7.stderr parsl.dataflow.dflow DEBUG: Task 1882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1882 has memoization hash 3ec3eaf8f4cb98d1edee580bf39ec216 parsl.dataflow.memoization INFO: Task 1882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6779b2056fefb2f3ee481b3e0a7586fc with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e86b24ae-78d6-480f-bb2c-b97a933c1f54_isrForCrosstalkSources_1031703_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e86b24ae-78d6-480f-bb2c-b97a933c1f54_isrForCrosstalkSources_1031703_13.stderr parsl.dataflow.dflow INFO: Parsl task 1882 try 0 launched on executor multi with executor id 1883 parsl.dataflow.dflow INFO: Standard output for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5a5ee25c-ef58-4371-bb7b-f7abf49a3ca2_isrForCrosstalkSources_1031714_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5a5ee25c-ef58-4371-bb7b-f7abf49a3ca2_isrForCrosstalkSources_1031714_61.stderr parsl.dataflow.dflow DEBUG: Task 1883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1883 has memoization hash 0cbc6258d184ebf91cabc744c0986631 parsl.dataflow.memoization INFO: Task 1883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3d3adbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1883 try 0 launched on executor multi with executor id 1884 parsl.dataflow.dflow INFO: Standard output for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/df87e422-1f4a-4fdd-88d5-d30d8b98f930_isrForCrosstalkSources_1031732_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/df87e422-1f4a-4fdd-88d5-d30d8b98f930_isrForCrosstalkSources_1031732_15.stderr parsl.dataflow.dflow DEBUG: Task 1884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1884 has memoization hash ad2556c3c39efe37d1d3efb2d0ba94c2 parsl.dataflow.memoization INFO: Task 1884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f4ae177e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1884 try 0 launched on executor multi with executor id 1885 parsl.dataflow.dflow INFO: Standard output for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/c7b1c589-39e4-4c6d-8cab-2a1c4812b4c0_isrForCrosstalkSources_1031701_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/c7b1c589-39e4-4c6d-8cab-2a1c4812b4c0_isrForCrosstalkSources_1031701_32.stderr parsl.dataflow.dflow DEBUG: Task 1885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1885 has memoization hash 3cc4fa8741e962e6a7ac9e4b2be2d71a parsl.dataflow.memoization INFO: Task 1885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1885 try 0 launched on executor multi with executor id 1886 parsl.dataflow.dflow INFO: Standard output for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/9a926a4e-7d77-4ffa-924d-e62c96cdf867_isrForCrosstalkSources_1031699_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/9a926a4e-7d77-4ffa-924d-e62c96cdf867_isrForCrosstalkSources_1031699_57.stderr parsl.dataflow.dflow DEBUG: Task 1886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1886 has memoization hash 5edd4e7cb990ee763b06265acd9de64e parsl.dataflow.memoization INFO: Task 1886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8734c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1886 try 0 launched on executor multi with executor id 1887 parsl.dataflow.dflow INFO: Standard output for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fc889e5f-124f-418f-8372-9230b29b0ba9_isrForCrosstalkSources_1031729_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fc889e5f-124f-418f-8372-9230b29b0ba9_isrForCrosstalkSources_1031729_10.stderr parsl.dataflow.dflow DEBUG: Task 1887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1887 has memoization hash c315952d9228f06d97cf9b0516a02943 parsl.dataflow.memoization INFO: Task 1887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f8732e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1887 try 0 launched on executor multi with executor id 1888 parsl.dataflow.dflow INFO: Standard output for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2622d988-79dd-44b2-8a53-41dbda1432cc_isrForCrosstalkSources_1031708_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2622d988-79dd-44b2-8a53-41dbda1432cc_isrForCrosstalkSources_1031708_13.stderr parsl.dataflow.dflow DEBUG: Task 1888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1888 has memoization hash 3ced045a7146ebb679f58d1536b649b4 parsl.dataflow.memoization INFO: Task 1888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1888 try 0 launched on executor multi with executor id 1889 parsl.dataflow.dflow INFO: Standard output for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2bcf4220-ae97-43cf-94b7-c4323acc79eb_isrForCrosstalkSources_1031703_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2bcf4220-ae97-43cf-94b7-c4323acc79eb_isrForCrosstalkSources_1031703_50.stderr parsl.dataflow.dflow DEBUG: Task 1889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1889 has memoization hash 759bdb2ae83d97e3fb83f6932ecd56a7 parsl.dataflow.memoization INFO: Task 1889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f873600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1889 try 0 launched on executor multi with executor id 1890 parsl.dataflow.dflow INFO: Standard output for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f5c8ac6f-d47a-46e1-b451-c484198fa128_isrForCrosstalkSources_1031732_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f5c8ac6f-d47a-46e1-b451-c484198fa128_isrForCrosstalkSources_1031732_23.stderr parsl.dataflow.dflow DEBUG: Task 1890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1890 has memoization hash 65618423145660d07a250ec40591fcae parsl.dataflow.memoization INFO: Task 1890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1890 try 0 launched on executor multi with executor id 1891 parsl.dataflow.dflow INFO: Standard output for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c9db02ca-b3cc-4f48-9d7a-c6a355fa9dfd_isrForCrosstalkSources_1031716_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c9db02ca-b3cc-4f48-9d7a-c6a355fa9dfd_isrForCrosstalkSources_1031716_12.stderr parsl.dataflow.dflow DEBUG: Task 1891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1891 has memoization hash 86790e45ce0d1f7750632f02bc503a8b parsl.dataflow.memoization INFO: Task 1891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1891 try 0 launched on executor multi with executor id 1892 parsl.dataflow.dflow INFO: Standard output for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/edf78914-0552-474c-9d2b-1c66fdf7ddd7_isrForCrosstalkSources_1031729_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/edf78914-0552-474c-9d2b-1c66fdf7ddd7_isrForCrosstalkSources_1031729_7.stderr parsl.dataflow.dflow DEBUG: Task 1892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1892 has memoization hash 55050ade6fe33f2e1e866e6a7cba9cc8 parsl.dataflow.memoization INFO: Task 1892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1892 try 0 launched on executor multi with executor id 1893 parsl.dataflow.dflow INFO: Standard output for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3296bd3e-3a85-4057-bc52-44a8e0f135af_isrForCrosstalkSources_1031729_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3296bd3e-3a85-4057-bc52-44a8e0f135af_isrForCrosstalkSources_1031729_62.stderr parsl.dataflow.dflow DEBUG: Task 1893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1893 has memoization hash 5e168256110c54d1878f26c9ed14812f parsl.dataflow.memoization INFO: Task 1893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1893 try 0 launched on executor multi with executor id 1894 parsl.dataflow.dflow INFO: Standard output for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/f96534e8-dc4e-4e46-8c29-00072fdfc60a_isrForCrosstalkSources_1031702_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/f96534e8-dc4e-4e46-8c29-00072fdfc60a_isrForCrosstalkSources_1031702_36.stderr parsl.dataflow.dflow DEBUG: Task 1894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1894 has memoization hash b9fffea673e2d5f38e82e260395182e7 parsl.dataflow.memoization INFO: Task 1894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1894 try 0 launched on executor multi with executor id 1895 parsl.dataflow.dflow INFO: Standard output for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ba9bc927-bf4a-4dee-935a-744e8c3ea8dc_isrForCrosstalkSources_1031709_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ba9bc927-bf4a-4dee-935a-744e8c3ea8dc_isrForCrosstalkSources_1031709_39.stderr parsl.dataflow.dflow DEBUG: Task 1895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1895 has memoization hash 2c7641d8a3103d96959726104ed169cb parsl.dataflow.memoization INFO: Task 1895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1895 try 0 launched on executor multi with executor id 1896 parsl.dataflow.dflow INFO: Standard output for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/3bfce2e5-bd57-4867-8002-d89e4252b536_isrForCrosstalkSources_1031726_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/3bfce2e5-bd57-4867-8002-d89e4252b536_isrForCrosstalkSources_1031726_25.stderr parsl.dataflow.dflow DEBUG: Task 1896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1896 has memoization hash c218090225e552c76182dada79baa1d9 parsl.dataflow.memoization INFO: Task 1896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1896 try 0 launched on executor multi with executor id 1897 parsl.dataflow.dflow INFO: Standard output for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/e286aec0-c46f-4d96-858b-96e909a4ab95_isrForCrosstalkSources_1031708_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/e286aec0-c46f-4d96-858b-96e909a4ab95_isrForCrosstalkSources_1031708_4.stderr parsl.dataflow.dflow DEBUG: Task 1897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1897 has memoization hash e3600eeb1b51dc18ac543e4b90085dea parsl.dataflow.memoization INFO: Task 1897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1897 try 0 launched on executor multi with executor id 1898 parsl.dataflow.dflow INFO: Standard output for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/5c951d22-66b4-461b-8623-7b3c86ed4790_isrForCrosstalkSources_1031722_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/5c951d22-66b4-461b-8623-7b3c86ed4790_isrForCrosstalkSources_1031722_62.stderr parsl.dataflow.dflow DEBUG: Task 1898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1898 has memoization hash 495735df98930a591b771373f3cca421 parsl.dataflow.memoization INFO: Task 1898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1898 try 0 launched on executor multi with executor id 1899 parsl.dataflow.dflow INFO: Standard output for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c7ae58d5-acb1-4ca2-bf20-e3226593333d_isrForCrosstalkSources_1031728_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c7ae58d5-acb1-4ca2-bf20-e3226593333d_isrForCrosstalkSources_1031728_55.stderr parsl.dataflow.dflow DEBUG: Task 1899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1899 has memoization hash a751a4d3cd3741ddf5fbe49078ba9efd parsl.dataflow.memoization INFO: Task 1899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1899 try 0 launched on executor multi with executor id 1900 parsl.dataflow.dflow INFO: Standard output for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7d547375-1f31-49f6-923d-4af9699848b1_isrForCrosstalkSources_1031702_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7d547375-1f31-49f6-923d-4af9699848b1_isrForCrosstalkSources_1031702_20.stderr parsl.dataflow.dflow DEBUG: Task 1900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1900 has memoization hash 36f6e621df5ea3ea338078e77a200056 parsl.dataflow.memoization INFO: Task 1900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1900 try 0 launched on executor multi with executor id 1901 parsl.dataflow.dflow INFO: Standard output for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/93ab53b8-a812-4709-ae5b-2f779af5c012_isrForCrosstalkSources_1031737_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/93ab53b8-a812-4709-ae5b-2f779af5c012_isrForCrosstalkSources_1031737_35.stderr parsl.dataflow.dflow DEBUG: Task 1901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1901 has memoization hash 3eff06dc89823cd5d9c25d92b87e5faa parsl.dataflow.memoization INFO: Task 1901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1901 try 0 launched on executor multi with executor id 1902 parsl.dataflow.dflow INFO: Standard output for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b336c411-a3de-4412-802d-0bd081bbb7c8_isrForCrosstalkSources_1031716_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b336c411-a3de-4412-802d-0bd081bbb7c8_isrForCrosstalkSources_1031716_45.stderr parsl.dataflow.dflow DEBUG: Task 1902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1902 has memoization hash 2231e01f69302e81f506953437e2b9aa parsl.dataflow.memoization INFO: Task 1902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1902 try 0 launched on executor multi with executor id 1903 parsl.dataflow.dflow INFO: Standard output for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f6de5add-c54a-4f7a-a461-8bcf1a96bc77_isrForCrosstalkSources_1031721_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f6de5add-c54a-4f7a-a461-8bcf1a96bc77_isrForCrosstalkSources_1031721_54.stderr parsl.dataflow.dflow DEBUG: Task 1903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1903 has memoization hash 76adfc94e0a9e3e30bab4e9c361f5276 parsl.dataflow.memoization INFO: Task 1903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1903 try 0 launched on executor multi with executor id 1904 parsl.dataflow.dflow INFO: Standard output for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/57bc79ea-b821-401f-a2e6-1a5a14599776_isrForCrosstalkSources_1031722_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/57bc79ea-b821-401f-a2e6-1a5a14599776_isrForCrosstalkSources_1031722_16.stderr parsl.dataflow.dflow DEBUG: Task 1904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1904 has memoization hash d2f27ef20ab0e1ec7b749c9889afc6a7 parsl.dataflow.memoization INFO: Task 1904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1904 try 0 launched on executor multi with executor id 1905 parsl.dataflow.dflow INFO: Standard output for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/ece6a0ca-1a07-4938-a229-c0824343ae89_isrForCrosstalkSources_1031704_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/ece6a0ca-1a07-4938-a229-c0824343ae89_isrForCrosstalkSources_1031704_6.stderr parsl.dataflow.dflow DEBUG: Task 1905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1905 has memoization hash 077078192b02ccb99981a71048782127 parsl.dataflow.memoization INFO: Task 1905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1905 try 0 launched on executor multi with executor id 1906 parsl.dataflow.dflow INFO: Standard output for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0f078bd4-a046-4fd9-bcd2-61cd85c5f9ba_isrForCrosstalkSources_1031714_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0f078bd4-a046-4fd9-bcd2-61cd85c5f9ba_isrForCrosstalkSources_1031714_1.stderr parsl.dataflow.dflow DEBUG: Task 1906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1906 has memoization hash 81896b0d7ab4644ea6609e0d757a8a27 parsl.dataflow.memoization INFO: Task 1906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1906 try 0 launched on executor multi with executor id 1907 parsl.dataflow.dflow INFO: Standard output for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bbfb01d1-c7da-4900-a2b8-9ee06209505b_isrForCrosstalkSources_1031729_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bbfb01d1-c7da-4900-a2b8-9ee06209505b_isrForCrosstalkSources_1031729_33.stderr parsl.dataflow.dflow DEBUG: Task 1907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1907 has memoization hash 3eb9692f5e7626a514eb2288e5416399 parsl.dataflow.memoization INFO: Task 1907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1907 try 0 launched on executor multi with executor id 1908 parsl.dataflow.dflow INFO: Standard output for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9fa72e42-7d99-4b60-afad-ca4743303d64_isrForCrosstalkSources_1031718_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9fa72e42-7d99-4b60-afad-ca4743303d64_isrForCrosstalkSources_1031718_27.stderr parsl.dataflow.dflow DEBUG: Task 1908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1908 has memoization hash 0c8fdbc56f3c1df0cc8f2c5b499665c0 parsl.dataflow.memoization INFO: Task 1908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1908 try 0 launched on executor multi with executor id 1909 parsl.dataflow.dflow INFO: Standard output for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/1eb7b5ad-c1d4-4211-9d30-fb0422cfa807_isrForCrosstalkSources_1031730_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/1eb7b5ad-c1d4-4211-9d30-fb0422cfa807_isrForCrosstalkSources_1031730_12.stderr parsl.dataflow.dflow DEBUG: Task 1909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1909 has memoization hash b354750ceaf52777d303a6f63f8c55ca parsl.dataflow.memoization INFO: Task 1909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1909 try 0 launched on executor multi with executor id 1910 parsl.dataflow.dflow INFO: Standard output for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/aa8eeb14-0fb7-42b8-af15-258aa8bea2ab_isrForCrosstalkSources_1031724_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/aa8eeb14-0fb7-42b8-af15-258aa8bea2ab_isrForCrosstalkSources_1031724_35.stderr parsl.dataflow.dflow DEBUG: Task 1910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1910 has memoization hash 1c566934ccf6d9eb85c2804fb45858a7 parsl.dataflow.memoization INFO: Task 1910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1910 try 0 launched on executor multi with executor id 1911 parsl.dataflow.dflow INFO: Standard output for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3d3bf068-abdd-4999-81c5-1d9373ce4bc0_isrForCrosstalkSources_1031719_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3d3bf068-abdd-4999-81c5-1d9373ce4bc0_isrForCrosstalkSources_1031719_2.stderr parsl.dataflow.dflow DEBUG: Task 1911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1911 has memoization hash 5acfe2a7ef24f0e6d12b8eace31630e3 parsl.dataflow.memoization INFO: Task 1911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1911 try 0 launched on executor multi with executor id 1912 parsl.dataflow.dflow INFO: Standard output for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/119048a3-6545-48f1-a35b-75e05240910e_isrForCrosstalkSources_1031722_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/119048a3-6545-48f1-a35b-75e05240910e_isrForCrosstalkSources_1031722_51.stderr parsl.dataflow.dflow DEBUG: Task 1912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1912 has memoization hash ea1c6dc5cc74d0bca51c302ab9228ac1 parsl.dataflow.memoization INFO: Task 1912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1912 try 0 launched on executor multi with executor id 1913 parsl.dataflow.dflow INFO: Standard output for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/5ca89d51-8543-4030-a45f-789c4aac7710_isrForCrosstalkSources_1031706_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/5ca89d51-8543-4030-a45f-789c4aac7710_isrForCrosstalkSources_1031706_17.stderr parsl.dataflow.dflow DEBUG: Task 1913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1913 has memoization hash 0d723223cf1046fce1697c4bfbbb43be parsl.dataflow.memoization INFO: Task 1913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1913 try 0 launched on executor multi with executor id 1914 parsl.dataflow.dflow INFO: Standard output for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/e7e6e2e6-8a1f-4917-adba-338796686fb8_isrForCrosstalkSources_1031734_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/e7e6e2e6-8a1f-4917-adba-338796686fb8_isrForCrosstalkSources_1031734_35.stderr parsl.dataflow.dflow DEBUG: Task 1914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1914 has memoization hash 2c3dd105ad234afb43ef7ab37f369442 parsl.dataflow.memoization INFO: Task 1914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1914 try 0 launched on executor multi with executor id 1915 parsl.dataflow.dflow INFO: Standard output for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1657c898-aa0f-44b6-8c1f-e5f82730d2fd_isrForCrosstalkSources_1031700_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1657c898-aa0f-44b6-8c1f-e5f82730d2fd_isrForCrosstalkSources_1031700_2.stderr parsl.dataflow.dflow DEBUG: Task 1915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1915 has memoization hash 703013c656b65b6bf3e3f8fb7b8dee39 parsl.dataflow.memoization INFO: Task 1915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1915 try 0 launched on executor multi with executor id 1916 parsl.dataflow.dflow INFO: Standard output for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/579352d8-dc53-4626-a09c-14a7f9d0a76e_isrForCrosstalkSources_1031720_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/579352d8-dc53-4626-a09c-14a7f9d0a76e_isrForCrosstalkSources_1031720_49.stderr parsl.dataflow.dflow DEBUG: Task 1916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1916 has memoization hash f3db489bce85869eeab24435feeedaaa parsl.dataflow.memoization INFO: Task 1916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1916 try 0 launched on executor multi with executor id 1917 parsl.dataflow.dflow INFO: Standard output for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/28129b60-160e-4f10-bdd5-ebcd9cb5c02c_isrForCrosstalkSources_1031702_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/28129b60-160e-4f10-bdd5-ebcd9cb5c02c_isrForCrosstalkSources_1031702_62.stderr parsl.dataflow.dflow DEBUG: Task 1917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1917 has memoization hash 029ebd05a3e8cad89b26506aa8564fe4 parsl.dataflow.memoization INFO: Task 1917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1917 try 0 launched on executor multi with executor id 1918 parsl.dataflow.dflow INFO: Standard output for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/53d8fb18-85c8-40a3-9f78-2b64cb70939a_isrForCrosstalkSources_1031730_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/53d8fb18-85c8-40a3-9f78-2b64cb70939a_isrForCrosstalkSources_1031730_30.stderr parsl.dataflow.dflow DEBUG: Task 1918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1918 has memoization hash e76a0b297ceacaf8ee905179e02df234 parsl.dataflow.memoization INFO: Task 1918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1918 try 0 launched on executor multi with executor id 1919 parsl.dataflow.dflow INFO: Standard output for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/eb01b66e-f3a5-48c6-a433-0c8489179263_isrForCrosstalkSources_1031723_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/eb01b66e-f3a5-48c6-a433-0c8489179263_isrForCrosstalkSources_1031723_25.stderr parsl.dataflow.dflow DEBUG: Task 1919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1919 has memoization hash bb661fb9b7f0c9ceed5c3adf9a7b75b3 parsl.dataflow.memoization INFO: Task 1919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 357272ce3a60ed8b870eadd0ce6d2688 with result from task 4 parsl.dataflow.dflow INFO: Parsl task 1919 try 0 launched on executor multi with executor id 1920 parsl.dataflow.dflow INFO: Standard output for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3bd2023b-3a21-4b82-ad1d-d8a2163cdcb0_isrForCrosstalkSources_1031713_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3bd2023b-3a21-4b82-ad1d-d8a2163cdcb0_isrForCrosstalkSources_1031713_31.stderr parsl.dataflow.dflow DEBUG: Task 1920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1920 has memoization hash 759e481feb9431c38cdc885b39235b3a parsl.dataflow.memoization INFO: Task 1920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1920 try 0 launched on executor multi with executor id 1921 parsl.dataflow.dflow INFO: Standard output for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/3bc050d3-d8cc-4a13-a502-01749d99c82a_isrForCrosstalkSources_1031734_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/3bc050d3-d8cc-4a13-a502-01749d99c82a_isrForCrosstalkSources_1031734_56.stderr parsl.dataflow.dflow DEBUG: Task 1921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1921 has memoization hash 2321f0e2bdd12fb8100b5b995575117a parsl.dataflow.memoization INFO: Task 1921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Parsl task 1921 try 0 launched on executor multi with executor id 1922 parsl.dataflow.dflow INFO: Standard output for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/a4c4994b-fe7b-4b3d-8d58-f84fa8c95ec4_isrForCrosstalkSources_1031714_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/a4c4994b-fe7b-4b3d-8d58-f84fa8c95ec4_isrForCrosstalkSources_1031714_19.stderr parsl.dataflow.dflow DEBUG: Task 1922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1922 has memoization hash 6aa11725271286e73f2b7d98ce69e6ef parsl.dataflow.memoization INFO: Task 1922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1922 try 0 launched on executor multi with executor id 1923 parsl.dataflow.dflow INFO: Standard output for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a12848d7-1ea7-4c68-b058-81552877f56e_isrForCrosstalkSources_1031723_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a12848d7-1ea7-4c68-b058-81552877f56e_isrForCrosstalkSources_1031723_58.stderr parsl.dataflow.dflow DEBUG: Task 1923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1923 has memoization hash 6757c6adb4e6a7181e52cdfcb6bb73ba parsl.dataflow.memoization INFO: Task 1923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/391214f3-0e3b-4e9d-8b44-4ddf28950b1f_isrForCrosstalkSources_1031732_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/391214f3-0e3b-4e9d-8b44-4ddf28950b1f_isrForCrosstalkSources_1031732_21.stderr parsl.dataflow.dflow INFO: Parsl task 1923 try 0 launched on executor multi with executor id 1924 parsl.dataflow.dflow INFO: Standard output for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/ce98becc-8bf2-4e4a-95aa-86e017124654_isrForCrosstalkSources_1031712_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/ce98becc-8bf2-4e4a-95aa-86e017124654_isrForCrosstalkSources_1031712_40.stderr parsl.dataflow.dflow DEBUG: Task 1924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1924 has memoization hash 10ea28955ff01f40b90948e365611b2f parsl.dataflow.memoization INFO: Task 1924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1924 try 0 launched on executor multi with executor id 1925 parsl.dataflow.dflow INFO: Standard output for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4ce9c7bd-1497-4359-8c69-69d36617de01_isrForCrosstalkSources_1031699_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4ce9c7bd-1497-4359-8c69-69d36617de01_isrForCrosstalkSources_1031699_51.stderr parsl.dataflow.dflow DEBUG: Task 1925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1925 has memoization hash e0b8cf09192bd4d75c6ddafbfdd904df parsl.dataflow.memoization INFO: Task 1925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:20 proc_lsst.multi:170 [INFO] [multi] job_id 17147955 proc_lsst.multi INFO: [multi] job_id 17147955 2024-03-17 04:02:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-17 04:02:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147955 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147955 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17147955 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-17 04:02:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-17 04:02:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-17 04:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1925 try 0 launched on executor multi with executor id 1926 parsl.dataflow.dflow INFO: Standard output for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/05fc06bb-94a3-4907-8e3f-3aafdfd590ad_isrForCrosstalkSources_1031727_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/05fc06bb-94a3-4907-8e3f-3aafdfd590ad_isrForCrosstalkSources_1031727_15.stderr parsl.dataflow.dflow DEBUG: Task 1926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1926 has memoization hash 899f1ff0722108b0bf5edc9376accc9f parsl.dataflow.memoization INFO: Task 1926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f948d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1926 try 0 launched on executor multi with executor id 1927 parsl.dataflow.dflow INFO: Standard output for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/7ebce880-feb9-46b9-9f46-fdfd8d8d8307_isrForCrosstalkSources_1031731_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/7ebce880-feb9-46b9-9f46-fdfd8d8d8307_isrForCrosstalkSources_1031731_53.stderr parsl.dataflow.dflow DEBUG: Task 1927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1927 has memoization hash 4b48fe45aa984114abe38dc393942075 parsl.dataflow.memoization INFO: Task 1927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f871300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1927 try 0 launched on executor multi with executor id 1928 parsl.dataflow.dflow INFO: Standard output for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e6ae05d3-17a8-4e2d-a07e-25f4d586274f_isrForCrosstalkSources_1031731_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e6ae05d3-17a8-4e2d-a07e-25f4d586274f_isrForCrosstalkSources_1031731_48.stderr parsl.dataflow.dflow DEBUG: Task 1928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1928 has memoization hash 05c653daaabe452682463fb291544d0c parsl.dataflow.memoization INFO: Task 1928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f870040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1928 try 0 launched on executor multi with executor id 1929 parsl.dataflow.dflow INFO: Standard output for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/54a71da7-0714-4db1-bb79-3be0357ecb57_isrForCrosstalkSources_1031714_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/54a71da7-0714-4db1-bb79-3be0357ecb57_isrForCrosstalkSources_1031714_17.stderr parsl.dataflow.dflow DEBUG: Task 1929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1929 has memoization hash a503283fa4631c1e86ef35c81e5d768a parsl.dataflow.memoization INFO: Task 1929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1929 try 0 launched on executor multi with executor id 1930 parsl.dataflow.dflow INFO: Standard output for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/731d8a82-db21-4e97-98ef-08fefec70424_isrForCrosstalkSources_1031728_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/731d8a82-db21-4e97-98ef-08fefec70424_isrForCrosstalkSources_1031728_50.stderr parsl.dataflow.dflow DEBUG: Task 1930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1930 has memoization hash db6d216eb7a23880f94b080ccb075836 parsl.dataflow.memoization INFO: Task 1930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1930 try 0 launched on executor multi with executor id 1931 parsl.dataflow.dflow INFO: Standard output for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e747061e-f259-4f62-b87c-38291543ddb4_isrForCrosstalkSources_1031706_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e747061e-f259-4f62-b87c-38291543ddb4_isrForCrosstalkSources_1031706_35.stderr parsl.dataflow.dflow DEBUG: Task 1931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1931 has memoization hash cf0c9f291bdf6b7d8f2f6a7a60d13bc6 parsl.dataflow.memoization INFO: Task 1931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1931 try 0 launched on executor multi with executor id 1932 parsl.dataflow.dflow INFO: Standard output for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/98a4861f-b62c-4d0e-9273-9b4b8ab09069_isrForCrosstalkSources_1031732_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/98a4861f-b62c-4d0e-9273-9b4b8ab09069_isrForCrosstalkSources_1031732_13.stderr parsl.dataflow.dflow DEBUG: Task 1932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1932 has memoization hash 685796a5dbb47ca81cf855d37123c404 parsl.dataflow.memoization INFO: Task 1932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1932 try 0 launched on executor multi with executor id 1933 parsl.dataflow.dflow INFO: Standard output for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/40f78ae6-e398-48ce-93ee-ca22bad91280_isrForCrosstalkSources_1031718_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/40f78ae6-e398-48ce-93ee-ca22bad91280_isrForCrosstalkSources_1031718_48.stderr parsl.dataflow.dflow DEBUG: Task 1933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1933 has memoization hash 4ead0cdfd0f9edd75e6a5f6e59dd401a parsl.dataflow.memoization INFO: Task 1933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1933 try 0 launched on executor multi with executor id 1934 parsl.dataflow.dflow INFO: Standard output for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/aa610763-9a0c-439e-9169-80a0d8cf1bea_isrForCrosstalkSources_1031710_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/aa610763-9a0c-439e-9169-80a0d8cf1bea_isrForCrosstalkSources_1031710_44.stderr parsl.dataflow.dflow DEBUG: Task 1934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1934 has memoization hash e4ae2e9a0ed95b8421f312fa1838f0b3 parsl.dataflow.memoization INFO: Task 1934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1934 try 0 launched on executor multi with executor id 1935 parsl.dataflow.dflow INFO: Standard output for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/446f5cc7-99d2-4a2f-bdab-48815d7d677a_isrForCrosstalkSources_1031725_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/446f5cc7-99d2-4a2f-bdab-48815d7d677a_isrForCrosstalkSources_1031725_8.stderr parsl.dataflow.dflow DEBUG: Task 1935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1935 has memoization hash 869cc229693dc1a22d894e8bdb3988ae parsl.dataflow.memoization INFO: Task 1935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1935 try 0 launched on executor multi with executor id 1936 parsl.dataflow.dflow INFO: Standard output for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/d869bee9-914d-4e1d-94fe-68ff365ca7ee_isrForCrosstalkSources_1031699_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/d869bee9-914d-4e1d-94fe-68ff365ca7ee_isrForCrosstalkSources_1031699_18.stderr parsl.dataflow.dflow DEBUG: Task 1936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1936 has memoization hash d8333eb468ac879f5589f87fa76bd411 parsl.dataflow.memoization INFO: Task 1936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1936 try 0 launched on executor multi with executor id 1937 parsl.dataflow.dflow INFO: Standard output for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/ec38687e-611e-4312-96a2-bf110d25c044_isrForCrosstalkSources_1031719_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/ec38687e-611e-4312-96a2-bf110d25c044_isrForCrosstalkSources_1031719_9.stderr parsl.dataflow.dflow DEBUG: Task 1937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1937 has memoization hash 0c909734a5e2cc487695ce0c587f2de6 parsl.dataflow.memoization INFO: Task 1937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1937 try 0 launched on executor multi with executor id 1938 parsl.dataflow.dflow INFO: Standard output for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/236fd25c-597d-40d0-a69f-826123177286_isrForCrosstalkSources_1031730_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/236fd25c-597d-40d0-a69f-826123177286_isrForCrosstalkSources_1031730_35.stderr parsl.dataflow.dflow DEBUG: Task 1938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1938 has memoization hash 554d6fd239be2b4f19eb839b033d027f parsl.dataflow.memoization INFO: Task 1938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1938 try 0 launched on executor multi with executor id 1939 parsl.dataflow.dflow INFO: Standard output for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/086812ca-ab0d-4aad-8881-bf666cf7c3a7_isrForCrosstalkSources_1031737_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/086812ca-ab0d-4aad-8881-bf666cf7c3a7_isrForCrosstalkSources_1031737_12.stderr parsl.dataflow.dflow DEBUG: Task 1939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1939 has memoization hash d7b64ba52e3ca5c91869b9b8e3b4b02f parsl.dataflow.memoization INFO: Task 1939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1939 try 0 launched on executor multi with executor id 1940 parsl.dataflow.dflow INFO: Standard output for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/29c83bc4-030b-474f-b4be-946b45ff3e8e_isrForCrosstalkSources_1031725_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/29c83bc4-030b-474f-b4be-946b45ff3e8e_isrForCrosstalkSources_1031725_20.stderr parsl.dataflow.dflow DEBUG: Task 1940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1940 has memoization hash dc2d582d0d3145d47c281e2ea00381dd parsl.dataflow.memoization INFO: Task 1940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1940 try 0 launched on executor multi with executor id 1941 parsl.dataflow.dflow INFO: Standard output for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/521a28c1-3075-486b-a77e-adccaedb75aa_isrForCrosstalkSources_1031733_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/521a28c1-3075-486b-a77e-adccaedb75aa_isrForCrosstalkSources_1031733_29.stderr parsl.dataflow.dflow DEBUG: Task 1941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1941 has memoization hash acdd642d210ca75a02f0071064a9eeec parsl.dataflow.memoization INFO: Task 1941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1941 try 0 launched on executor multi with executor id 1942 parsl.dataflow.dflow INFO: Standard output for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c4e6219f-6f16-4223-81da-f11eddb747cf_isrForCrosstalkSources_1031719_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c4e6219f-6f16-4223-81da-f11eddb747cf_isrForCrosstalkSources_1031719_34.stderr parsl.dataflow.dflow DEBUG: Task 1942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1942 has memoization hash 3381d205ef80b6f11cce4c5716178d85 parsl.dataflow.memoization INFO: Task 1942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1942 try 0 launched on executor multi with executor id 1943 parsl.dataflow.dflow INFO: Standard output for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/3039e25f-98c1-41f7-a315-eb5efd184952_isrForCrosstalkSources_1031733_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/3039e25f-98c1-41f7-a315-eb5efd184952_isrForCrosstalkSources_1031733_10.stderr parsl.dataflow.dflow DEBUG: Task 1943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1943 has memoization hash d73c7395f99d293ed76e1de7da48db07 parsl.dataflow.memoization INFO: Task 1943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1943 try 0 launched on executor multi with executor id 1944 parsl.dataflow.dflow INFO: Standard output for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2f6b004b-c3d4-46f5-9752-24a6fd18c16e_isrForCrosstalkSources_1031731_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2f6b004b-c3d4-46f5-9752-24a6fd18c16e_isrForCrosstalkSources_1031731_9.stderr parsl.dataflow.dflow DEBUG: Task 1944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1944 has memoization hash ca4700f8a4465988a0b7453d9b08b125 parsl.dataflow.memoization INFO: Task 1944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1944 try 0 launched on executor multi with executor id 1945 parsl.dataflow.dflow INFO: Standard output for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/2eef6316-0f4b-47b1-9604-2540054afb96_isrForCrosstalkSources_1031705_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/2eef6316-0f4b-47b1-9604-2540054afb96_isrForCrosstalkSources_1031705_31.stderr parsl.dataflow.dflow DEBUG: Task 1945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1945 has memoization hash 232e0b21ab4834950c9f0caf7fe282d4 parsl.dataflow.memoization INFO: Task 1945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1945 try 0 launched on executor multi with executor id 1946 parsl.dataflow.dflow INFO: Standard output for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/97265be7-e3c6-45a1-a50a-f131fdab0c00_isrForCrosstalkSources_1031731_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/97265be7-e3c6-45a1-a50a-f131fdab0c00_isrForCrosstalkSources_1031731_44.stderr parsl.dataflow.dflow DEBUG: Task 1946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1946 has memoization hash ecf0340b491e42cd0c481492ffbc3231 parsl.dataflow.memoization INFO: Task 1946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1946 try 0 launched on executor multi with executor id 1947 parsl.dataflow.dflow INFO: Standard output for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/6650b27f-4d16-47e2-a897-15ae52043a06_isrForCrosstalkSources_1031734_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/6650b27f-4d16-47e2-a897-15ae52043a06_isrForCrosstalkSources_1031734_21.stderr parsl.dataflow.dflow DEBUG: Task 1947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1947 has memoization hash 9ddce6e7aaef3a20c0b74cd204c403f2 parsl.dataflow.memoization INFO: Task 1947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1947 try 0 launched on executor multi with executor id 1948 parsl.dataflow.dflow INFO: Standard output for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/7cc8d542-13de-43e9-a636-1cccb69839d0_isrForCrosstalkSources_1031717_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/7cc8d542-13de-43e9-a636-1cccb69839d0_isrForCrosstalkSources_1031717_60.stderr parsl.dataflow.dflow DEBUG: Task 1948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1948 has memoization hash 137aed51bdefe056d1c62fbe3ac9dd5f parsl.dataflow.memoization INFO: Task 1948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1948 try 0 launched on executor multi with executor id 1949 parsl.dataflow.dflow INFO: Standard output for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5bbdaefd-4636-46bd-883f-2afe2a6de33a_isrForCrosstalkSources_1031701_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5bbdaefd-4636-46bd-883f-2afe2a6de33a_isrForCrosstalkSources_1031701_18.stderr parsl.dataflow.dflow DEBUG: Task 1949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1949 has memoization hash a5438856df981f799121175dda7bfb9e parsl.dataflow.memoization INFO: Task 1949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1949 try 0 launched on executor multi with executor id 1950 parsl.dataflow.dflow INFO: Standard output for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5dbd1431-ad2a-4357-925a-7cfe600a3e69_isrForCrosstalkSources_1031720_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5dbd1431-ad2a-4357-925a-7cfe600a3e69_isrForCrosstalkSources_1031720_23.stderr parsl.dataflow.dflow DEBUG: Task 1950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1950 has memoization hash f7a318e3f2dc60b82b516b269413f9d0 parsl.dataflow.memoization INFO: Task 1950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1950 try 0 launched on executor multi with executor id 1951 parsl.dataflow.dflow INFO: Standard output for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/9dbb4d1b-3a3a-494a-8063-4908e00daec5_isrForCrosstalkSources_1031734_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/9dbb4d1b-3a3a-494a-8063-4908e00daec5_isrForCrosstalkSources_1031734_45.stderr parsl.dataflow.dflow DEBUG: Task 1951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1951 has memoization hash 91f8f4db5d4adaefed5bf4adba807e64 parsl.dataflow.memoization INFO: Task 1951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1951 try 0 launched on executor multi with executor id 1952 parsl.dataflow.dflow INFO: Standard output for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/21ce9300-2fae-4b8a-9ae4-63c195f8dce6_isrForCrosstalkSources_1031711_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/21ce9300-2fae-4b8a-9ae4-63c195f8dce6_isrForCrosstalkSources_1031711_12.stderr parsl.dataflow.dflow DEBUG: Task 1952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1952 has memoization hash e03c1123ac0c5f23b77117a7951730cf parsl.dataflow.memoization INFO: Task 1952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1952 try 0 launched on executor multi with executor id 1953 parsl.dataflow.dflow INFO: Standard output for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/0fd57141-ad72-4d4c-858a-4f49ceda2edd_isrForCrosstalkSources_1031701_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/0fd57141-ad72-4d4c-858a-4f49ceda2edd_isrForCrosstalkSources_1031701_5.stderr parsl.dataflow.dflow DEBUG: Task 1953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1953 has memoization hash 7cedbfe1454da755e2dc80106e28f9a6 parsl.dataflow.memoization INFO: Task 1953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1953 try 0 launched on executor multi with executor id 1954 parsl.dataflow.dflow INFO: Standard output for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c5b789ce-f159-4fee-a1d1-227b5f8879a1_isrForCrosstalkSources_1031708_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c5b789ce-f159-4fee-a1d1-227b5f8879a1_isrForCrosstalkSources_1031708_14.stderr parsl.dataflow.dflow DEBUG: Task 1954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1954 has memoization hash 1fb1204515559902af6d8e61cfda6525 parsl.dataflow.memoization INFO: Task 1954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1954 try 0 launched on executor multi with executor id 1955 parsl.dataflow.dflow INFO: Standard output for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/5168fc8a-dead-43ef-9709-29aaeddfd9bb_isrForCrosstalkSources_1031713_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/5168fc8a-dead-43ef-9709-29aaeddfd9bb_isrForCrosstalkSources_1031713_17.stderr parsl.dataflow.dflow DEBUG: Task 1955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1955 has memoization hash a333a5fc33699f81ea21d14b87b3fc84 parsl.dataflow.memoization INFO: Task 1955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1955 try 0 launched on executor multi with executor id 1956 parsl.dataflow.dflow INFO: Standard output for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c02374f4-42a6-4d3f-b6b1-dc8ba67d6f23_isrForCrosstalkSources_1031729_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c02374f4-42a6-4d3f-b6b1-dc8ba67d6f23_isrForCrosstalkSources_1031729_48.stderr parsl.dataflow.dflow DEBUG: Task 1956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1956 has memoization hash 0aad658006a8994ccf751b6a43d0050f parsl.dataflow.memoization INFO: Task 1956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1956 try 0 launched on executor multi with executor id 1957 parsl.dataflow.dflow INFO: Standard output for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/31b4e2f2-2529-41d6-b43f-6621826a0a9a_isrForCrosstalkSources_1031718_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/31b4e2f2-2529-41d6-b43f-6621826a0a9a_isrForCrosstalkSources_1031718_47.stderr parsl.dataflow.dflow DEBUG: Task 1957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1957 has memoization hash bfc3a0740ef371c82ab77ede0ac29909 parsl.dataflow.memoization INFO: Task 1957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1957 try 0 launched on executor multi with executor id 1958 parsl.dataflow.dflow INFO: Standard output for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/afca63e1-fa5c-4e0d-a2d1-1bf6253f68c3_isrForCrosstalkSources_1031704_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/afca63e1-fa5c-4e0d-a2d1-1bf6253f68c3_isrForCrosstalkSources_1031704_15.stderr parsl.dataflow.dflow DEBUG: Task 1958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1958 has memoization hash 312d031a8fb11c989d8ee1daaae57f7e parsl.dataflow.memoization INFO: Task 1958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1958 try 0 launched on executor multi with executor id 1959 parsl.dataflow.dflow INFO: Standard output for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/8d7621c8-adc4-404c-a1e6-e62ede0b36d8_isrForCrosstalkSources_1031731_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/8d7621c8-adc4-404c-a1e6-e62ede0b36d8_isrForCrosstalkSources_1031731_45.stderr parsl.dataflow.dflow DEBUG: Task 1959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1959 has memoization hash 702c9241f3bffb957a52453378c5a68b parsl.dataflow.memoization INFO: Task 1959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1959 try 0 launched on executor multi with executor id 1960 parsl.dataflow.dflow INFO: Standard output for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0327063a-fa4d-4033-8165-b729e9e777c6_isrForCrosstalkSources_1031705_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0327063a-fa4d-4033-8165-b729e9e777c6_isrForCrosstalkSources_1031705_44.stderr parsl.dataflow.dflow DEBUG: Task 1960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1960 has memoization hash 317bae838117e9d5a42f011b253701d7 parsl.dataflow.memoization INFO: Task 1960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1960 try 0 launched on executor multi with executor id 1961 parsl.dataflow.dflow INFO: Standard output for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6814d677-9239-4c5b-8368-612c71f6566c_isrForCrosstalkSources_1031711_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6814d677-9239-4c5b-8368-612c71f6566c_isrForCrosstalkSources_1031711_24.stderr parsl.dataflow.dflow DEBUG: Task 1961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1961 has memoization hash 16ecf20f57b8861a7a61ec788251bbf0 parsl.dataflow.memoization INFO: Task 1961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1961 try 0 launched on executor multi with executor id 1962 parsl.dataflow.dflow INFO: Standard output for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/97114f2d-f8ab-4c6e-8315-7c0c89d495cb_isrForCrosstalkSources_1031734_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/97114f2d-f8ab-4c6e-8315-7c0c89d495cb_isrForCrosstalkSources_1031734_28.stderr parsl.dataflow.dflow DEBUG: Task 1962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1962 has memoization hash 433645a99d6ee2487d414b404d4aac6e parsl.dataflow.memoization INFO: Task 1962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1962 try 0 launched on executor multi with executor id 1963 parsl.dataflow.dflow INFO: Standard output for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/898b2c30-8e40-47ae-aa31-1f2c45753bcf_isrForCrosstalkSources_1031730_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/898b2c30-8e40-47ae-aa31-1f2c45753bcf_isrForCrosstalkSources_1031730_55.stderr parsl.dataflow.dflow DEBUG: Task 1963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1963 has memoization hash f5f81a8ad700b3d6373bf37514af8308 parsl.dataflow.memoization INFO: Task 1963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1963 try 0 launched on executor multi with executor id 1964 parsl.dataflow.dflow INFO: Standard output for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/db918598-354c-48a7-ad69-29876b9793f4_isrForCrosstalkSources_1031724_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/db918598-354c-48a7-ad69-29876b9793f4_isrForCrosstalkSources_1031724_11.stderr parsl.dataflow.dflow DEBUG: Task 1964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1964 has memoization hash a530f8a00cc3961947020e89be330119 parsl.dataflow.memoization INFO: Task 1964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1964 try 0 launched on executor multi with executor id 1965 parsl.dataflow.dflow INFO: Standard output for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2a0a9efc-d769-4519-84ef-ae6d3dc7b09b_isrForCrosstalkSources_1031723_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2a0a9efc-d769-4519-84ef-ae6d3dc7b09b_isrForCrosstalkSources_1031723_32.stderr parsl.dataflow.dflow DEBUG: Task 1965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1965 has memoization hash 647177509c5d3f20b704f0c5ebfe843a parsl.dataflow.memoization INFO: Task 1965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1965 try 0 launched on executor multi with executor id 1966 parsl.dataflow.dflow INFO: Standard output for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/4685e0a9-c44c-4032-87e6-24e9732b37d5_isrForCrosstalkSources_1031730_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/4685e0a9-c44c-4032-87e6-24e9732b37d5_isrForCrosstalkSources_1031730_20.stderr parsl.dataflow.dflow DEBUG: Task 1966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1966 has memoization hash 36fa9d8063e82edc2fb476578682146e parsl.dataflow.memoization INFO: Task 1966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1966 try 0 launched on executor multi with executor id 1967 parsl.dataflow.dflow INFO: Standard output for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/837b41e9-5d6d-4982-82d9-1226e38a8d4c_isrForCrosstalkSources_1031704_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/837b41e9-5d6d-4982-82d9-1226e38a8d4c_isrForCrosstalkSources_1031704_25.stderr parsl.dataflow.dflow DEBUG: Task 1967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1967 has memoization hash eecb2fbcde0b86f84f6cb1a6fff6bc98 parsl.dataflow.memoization INFO: Task 1967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1967 try 0 launched on executor multi with executor id 1968 parsl.dataflow.dflow INFO: Standard output for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/edb6cc17-5b03-4e3a-a9d7-09c657e9a772_isrForCrosstalkSources_1031713_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/edb6cc17-5b03-4e3a-a9d7-09c657e9a772_isrForCrosstalkSources_1031713_36.stderr parsl.dataflow.dflow DEBUG: Task 1968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1968 has memoization hash d3728d9d29487e7a555ce59d40d6d490 parsl.dataflow.memoization INFO: Task 1968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1968 try 0 launched on executor multi with executor id 1969 parsl.dataflow.dflow INFO: Standard output for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c598bdf5-69f1-49e9-8000-1c87d7d1ca08_isrForCrosstalkSources_1031734_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c598bdf5-69f1-49e9-8000-1c87d7d1ca08_isrForCrosstalkSources_1031734_7.stderr parsl.dataflow.dflow DEBUG: Task 1969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1969 has memoization hash 37174c5efa547a0c7c3bee58c96af44f parsl.dataflow.memoization INFO: Task 1969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1969 try 0 launched on executor multi with executor id 1970 parsl.dataflow.dflow INFO: Standard output for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/8899245d-46c3-42fc-8e90-bce0ddf9a188_isrForCrosstalkSources_1031715_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/8899245d-46c3-42fc-8e90-bce0ddf9a188_isrForCrosstalkSources_1031715_33.stderr parsl.dataflow.dflow DEBUG: Task 1970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1970 has memoization hash 34ba998923222c39ddecdeb3d1038bfe parsl.dataflow.memoization INFO: Task 1970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1970 try 0 launched on executor multi with executor id 1971 parsl.dataflow.dflow INFO: Standard output for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/77819fba-e673-43a1-9a38-f21c9bd45c66_isrForCrosstalkSources_1031734_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/77819fba-e673-43a1-9a38-f21c9bd45c66_isrForCrosstalkSources_1031734_30.stderr parsl.dataflow.dflow DEBUG: Task 1971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1971 has memoization hash 7f996320fc61138f769bfcb1e611ece3 parsl.dataflow.memoization INFO: Task 1971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1971 try 0 launched on executor multi with executor id 1972 parsl.dataflow.dflow INFO: Standard output for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8987509d-b10b-47ca-a2c5-d92bb7dd9b1d_isrForCrosstalkSources_1031699_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8987509d-b10b-47ca-a2c5-d92bb7dd9b1d_isrForCrosstalkSources_1031699_50.stderr parsl.dataflow.dflow DEBUG: Task 1972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1972 has memoization hash 8f88719c6fb76bbbbfe52b9c9f55429e parsl.dataflow.memoization INFO: Task 1972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1972 try 0 launched on executor multi with executor id 1973 parsl.dataflow.dflow INFO: Standard output for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d564906b-57de-49d3-8636-638bd757cac6_isrForCrosstalkSources_1031711_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d564906b-57de-49d3-8636-638bd757cac6_isrForCrosstalkSources_1031711_56.stderr parsl.dataflow.dflow DEBUG: Task 1973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1973 has memoization hash f15d33265415bfe9b9431b4fe2892d8e parsl.dataflow.memoization INFO: Task 1973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1973 try 0 launched on executor multi with executor id 1974 parsl.dataflow.dflow INFO: Standard output for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/70e740cb-16cf-4b9a-94b0-f33cd5525cb5_isrForCrosstalkSources_1031737_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/70e740cb-16cf-4b9a-94b0-f33cd5525cb5_isrForCrosstalkSources_1031737_28.stderr parsl.dataflow.dflow DEBUG: Task 1974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1974 has memoization hash 7217245593fafe75247a6456cdddf982 parsl.dataflow.memoization INFO: Task 1974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1974 try 0 launched on executor multi with executor id 1975 parsl.dataflow.dflow INFO: Standard output for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/937037fb-2cc1-42f6-8761-8d9dffbc61d3_isrForCrosstalkSources_1031711_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/937037fb-2cc1-42f6-8761-8d9dffbc61d3_isrForCrosstalkSources_1031711_6.stderr parsl.dataflow.dflow DEBUG: Task 1975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1975 has memoization hash 89478b60b642466f60e0a9189e93c819 parsl.dataflow.memoization INFO: Task 1975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1975 try 0 launched on executor multi with executor id 1976 parsl.dataflow.dflow INFO: Standard output for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7d325956-bea5-4a0a-b38f-066f82c8e93b_isrForCrosstalkSources_1031732_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7d325956-bea5-4a0a-b38f-066f82c8e93b_isrForCrosstalkSources_1031732_50.stderr parsl.dataflow.dflow DEBUG: Task 1976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1976 has memoization hash 92c65936f006eecaa8140f13030328fb parsl.dataflow.memoization INFO: Task 1976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1976 try 0 launched on executor multi with executor id 1977 parsl.dataflow.dflow INFO: Standard output for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/257064e0-8eb0-41e4-b36d-d18eef0caa84_isrForCrosstalkSources_1031736_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/257064e0-8eb0-41e4-b36d-d18eef0caa84_isrForCrosstalkSources_1031736_58.stderr parsl.dataflow.dflow DEBUG: Task 1977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1977 has memoization hash bc11ec9d9adb8e3ec2ec328872d81c49 parsl.dataflow.memoization INFO: Task 1977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1977 try 0 launched on executor multi with executor id 1978 parsl.dataflow.dflow INFO: Standard output for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/32307ce9-f935-4382-9c9f-869446ae1d92_isrForCrosstalkSources_1031736_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/32307ce9-f935-4382-9c9f-869446ae1d92_isrForCrosstalkSources_1031736_48.stderr parsl.dataflow.dflow DEBUG: Task 1978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1978 has memoization hash 7f0098cdc475fd90b4f0a5a949f355c9 parsl.dataflow.memoization INFO: Task 1978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1978 try 0 launched on executor multi with executor id 1979 parsl.dataflow.dflow INFO: Standard output for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/0c2329cb-7e1c-4682-84ea-8573e7f105a6_isrForCrosstalkSources_1031733_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/0c2329cb-7e1c-4682-84ea-8573e7f105a6_isrForCrosstalkSources_1031733_62.stderr parsl.dataflow.dflow DEBUG: Task 1979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1979 has memoization hash d9fad07e759a0c93fbfe7b9801999e18 parsl.dataflow.memoization INFO: Task 1979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1979 try 0 launched on executor multi with executor id 1980 parsl.dataflow.dflow INFO: Standard output for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e434a537-7ab7-41bd-9fa6-a0f73180aba3_isrForCrosstalkSources_1031727_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e434a537-7ab7-41bd-9fa6-a0f73180aba3_isrForCrosstalkSources_1031727_22.stderr parsl.dataflow.dflow DEBUG: Task 1980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1980 has memoization hash b9bef014b5b96e22cad2375c39fdc00a parsl.dataflow.memoization INFO: Task 1980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1980 try 0 launched on executor multi with executor id 1981 parsl.dataflow.dflow INFO: Standard output for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/0bdee7f5-052a-4133-b15f-ad542aa2f02b_isrForCrosstalkSources_1031704_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/0bdee7f5-052a-4133-b15f-ad542aa2f02b_isrForCrosstalkSources_1031704_2.stderr parsl.dataflow.dflow DEBUG: Task 1981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1981 has memoization hash ec8ae11abbc9721862755d9ac78d4e09 parsl.dataflow.memoization INFO: Task 1981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1981 try 0 launched on executor multi with executor id 1982 parsl.dataflow.dflow INFO: Standard output for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/0a514de3-255c-4f21-9632-565287aac0be_isrForCrosstalkSources_1031718_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/0a514de3-255c-4f21-9632-565287aac0be_isrForCrosstalkSources_1031718_1.stderr parsl.dataflow.dflow DEBUG: Task 1982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1982 has memoization hash dbc2f214b5e86f2b22b9323ecece822e parsl.dataflow.memoization INFO: Task 1982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1982 try 0 launched on executor multi with executor id 1983 parsl.dataflow.dflow INFO: Standard output for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8bd532d7-997a-47c7-bed5-0306c162db76_isrForCrosstalkSources_1031736_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8bd532d7-997a-47c7-bed5-0306c162db76_isrForCrosstalkSources_1031736_7.stderr parsl.dataflow.dflow DEBUG: Task 1983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1983 has memoization hash 8668888dfec77d26c59defccbfbd005a parsl.dataflow.memoization INFO: Task 1983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1983 try 0 launched on executor multi with executor id 1984 parsl.dataflow.dflow INFO: Standard output for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c87eac10-473e-467b-96ea-6c1f42c265c8_isrForCrosstalkSources_1031716_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c87eac10-473e-467b-96ea-6c1f42c265c8_isrForCrosstalkSources_1031716_49.stderr parsl.dataflow.dflow DEBUG: Task 1984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1984 has memoization hash fd6e8c6b7bdcb7e1dbfb1d520d241a91 parsl.dataflow.memoization INFO: Task 1984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1984 try 0 launched on executor multi with executor id 1985 parsl.dataflow.dflow INFO: Standard output for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/46b1d99f-26df-428e-a981-3a8289c9c554_isrForCrosstalkSources_1031716_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/46b1d99f-26df-428e-a981-3a8289c9c554_isrForCrosstalkSources_1031716_33.stderr parsl.dataflow.dflow DEBUG: Task 1985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1985 has memoization hash c7aaf36732cf8a5050e6a84d343dc311 parsl.dataflow.memoization INFO: Task 1985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1985 try 0 launched on executor multi with executor id 1986 parsl.dataflow.dflow INFO: Standard output for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/9dfe3a6c-8bbf-4bdf-bcf4-449de721dabd_isrForCrosstalkSources_1031712_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/9dfe3a6c-8bbf-4bdf-bcf4-449de721dabd_isrForCrosstalkSources_1031712_14.stderr parsl.dataflow.dflow DEBUG: Task 1986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1986 has memoization hash 8f9e5101bb8cb6031f55c66bb0cc393f parsl.dataflow.memoization INFO: Task 1986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1986 try 0 launched on executor multi with executor id 1987 parsl.dataflow.dflow INFO: Standard output for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/1c2d1963-0a3d-4852-8d9a-616ae19d374d_isrForCrosstalkSources_1031712_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/1c2d1963-0a3d-4852-8d9a-616ae19d374d_isrForCrosstalkSources_1031712_35.stderr parsl.dataflow.dflow DEBUG: Task 1987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1987 has memoization hash 88d36bc4a70e6d84548294135fbdf174 parsl.dataflow.memoization INFO: Task 1987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1987 try 0 launched on executor multi with executor id 1988 parsl.dataflow.dflow INFO: Standard output for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/19691c9c-0001-432a-acb5-86f9b162c091_isrForCrosstalkSources_1031736_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/19691c9c-0001-432a-acb5-86f9b162c091_isrForCrosstalkSources_1031736_35.stderr parsl.dataflow.dflow DEBUG: Task 1988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1988 has memoization hash 400e3875064e474c8320a1448f661090 parsl.dataflow.memoization INFO: Task 1988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1988 try 0 launched on executor multi with executor id 1989 parsl.dataflow.dflow INFO: Standard output for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/74634bb6-9dee-49f1-9c37-c01526490a44_isrForCrosstalkSources_1031710_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/74634bb6-9dee-49f1-9c37-c01526490a44_isrForCrosstalkSources_1031710_4.stderr parsl.dataflow.dflow DEBUG: Task 1989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1989 has memoization hash bfb0e942311d5a4a0b869e629529ba9a parsl.dataflow.memoization INFO: Task 1989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1989 try 0 launched on executor multi with executor id 1990 parsl.dataflow.dflow INFO: Standard output for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ed81e890-15f0-46ca-b604-2d7ac75d835f_isrForCrosstalkSources_1031713_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ed81e890-15f0-46ca-b604-2d7ac75d835f_isrForCrosstalkSources_1031713_8.stderr parsl.dataflow.dflow DEBUG: Task 1990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1990 has memoization hash 32be1e2361b2cc81ad57b48ca4b69106 parsl.dataflow.memoization INFO: Task 1990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1990 try 0 launched on executor multi with executor id 1991 parsl.dataflow.dflow INFO: Standard output for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/2fc98b34-f312-4f6c-9c16-1cc564a57e81_isrForCrosstalkSources_1031710_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/2fc98b34-f312-4f6c-9c16-1cc564a57e81_isrForCrosstalkSources_1031710_26.stderr parsl.dataflow.dflow DEBUG: Task 1991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1991 has memoization hash c96b6d34045de2a52adceabde9efbe40 parsl.dataflow.memoization INFO: Task 1991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1991 try 0 launched on executor multi with executor id 1992 parsl.dataflow.dflow INFO: Standard output for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/7ab70a61-b3ed-4b06-b652-0faf02fd419e_isrForCrosstalkSources_1031733_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/7ab70a61-b3ed-4b06-b652-0faf02fd419e_isrForCrosstalkSources_1031733_51.stderr parsl.dataflow.dflow DEBUG: Task 1992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1992 has memoization hash 053b81029a2eb459cbd80730b5c1a7d6 parsl.dataflow.memoization INFO: Task 1992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1992 try 0 launched on executor multi with executor id 1993 parsl.dataflow.dflow INFO: Standard output for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d5c09e2a-0f6f-4e4a-90d6-59ccebb6076a_isrForCrosstalkSources_1031709_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d5c09e2a-0f6f-4e4a-90d6-59ccebb6076a_isrForCrosstalkSources_1031709_62.stderr parsl.dataflow.dflow DEBUG: Task 1993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1993 has memoization hash a5aa78b4f22d7fb6df5ded98e31bf4e2 parsl.dataflow.memoization INFO: Task 1993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7700e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1993 try 0 launched on executor multi with executor id 1994 parsl.dataflow.dflow INFO: Standard output for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/acb19e96-f2a2-4133-b441-04039725502d_isrForCrosstalkSources_1031709_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/acb19e96-f2a2-4133-b441-04039725502d_isrForCrosstalkSources_1031709_48.stderr parsl.dataflow.dflow DEBUG: Task 1994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1994 has memoization hash 6308b03a31aa3e01531627dc7dc4a463 parsl.dataflow.memoization INFO: Task 1994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1994 try 0 launched on executor multi with executor id 1995 parsl.dataflow.dflow INFO: Standard output for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/bb2fe940-a386-4c28-926a-8f90f8c1b884_isrForCrosstalkSources_1031712_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/bb2fe940-a386-4c28-926a-8f90f8c1b884_isrForCrosstalkSources_1031712_34.stderr parsl.dataflow.dflow DEBUG: Task 1995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1995 has memoization hash 71c6197a2f9be4c0013e46e57e3280ec parsl.dataflow.memoization INFO: Task 1995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7702c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1995 try 0 launched on executor multi with executor id 1996 parsl.dataflow.dflow INFO: Standard output for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/e4b24a93-b364-4717-b7f2-21b468cf8107_isrForCrosstalkSources_1031721_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/e4b24a93-b364-4717-b7f2-21b468cf8107_isrForCrosstalkSources_1031721_29.stderr parsl.dataflow.dflow DEBUG: Task 1996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1996 has memoization hash 238c3c0f96eca552af954f8ec07dabd6 parsl.dataflow.memoization INFO: Task 1996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1996 try 0 launched on executor multi with executor id 1997 parsl.dataflow.dflow INFO: Standard output for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/cfe0afc3-e682-43dd-8451-6706b9a7fae0_isrForCrosstalkSources_1031713_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/cfe0afc3-e682-43dd-8451-6706b9a7fae0_isrForCrosstalkSources_1031713_61.stderr parsl.dataflow.dflow DEBUG: Task 1997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1997 has memoization hash aa21c727fc48add56552d0051e7aeb47 parsl.dataflow.memoization INFO: Task 1997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7704a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1997 try 0 launched on executor multi with executor id 1998 parsl.dataflow.dflow INFO: Standard output for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/97cf1437-6dc4-432e-9165-f367aa53e068_isrForCrosstalkSources_1031736_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/97cf1437-6dc4-432e-9165-f367aa53e068_isrForCrosstalkSources_1031736_33.stderr parsl.dataflow.dflow DEBUG: Task 1998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1998 has memoization hash 6e4706d5eb4311c5b2d511e0668328b4 parsl.dataflow.memoization INFO: Task 1998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1998 try 0 launched on executor multi with executor id 1999 parsl.dataflow.dflow INFO: Standard output for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8cc1fb31-42e9-4f1a-85f3-86657d55010c_isrForCrosstalkSources_1031699_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8cc1fb31-42e9-4f1a-85f3-86657d55010c_isrForCrosstalkSources_1031699_21.stderr parsl.dataflow.dflow DEBUG: Task 1999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1999 has memoization hash 1710616cfa82be81430d85426b084c45 parsl.dataflow.memoization INFO: Task 1999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1999 try 0 launched on executor multi with executor id 2000 parsl.dataflow.dflow INFO: Standard output for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/a64ab40d-a79f-4ce5-8bdf-54b987a0d2ae_isrForCrosstalkSources_1031700_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/a64ab40d-a79f-4ce5-8bdf-54b987a0d2ae_isrForCrosstalkSources_1031700_27.stderr parsl.dataflow.dflow DEBUG: Task 2000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2000 has memoization hash e0e23bac909f807f5dc7b669915ebcde parsl.dataflow.memoization INFO: Task 2000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7705e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2000 try 0 launched on executor multi with executor id 2001 parsl.dataflow.dflow INFO: Standard output for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/d24b61e8-3f54-43aa-ac43-bad0dacbfc7d_isrForCrosstalkSources_1031736_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/d24b61e8-3f54-43aa-ac43-bad0dacbfc7d_isrForCrosstalkSources_1031736_51.stderr parsl.dataflow.dflow DEBUG: Task 2001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2001 has memoization hash fb25d8306394e4189cadacdfef74ed2b parsl.dataflow.memoization INFO: Task 2001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2001 try 0 launched on executor multi with executor id 2002 parsl.dataflow.dflow INFO: Standard output for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/147107b4-2966-45c6-8be7-daffad5e4bd0_isrForCrosstalkSources_1031720_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/147107b4-2966-45c6-8be7-daffad5e4bd0_isrForCrosstalkSources_1031720_7.stderr parsl.dataflow.dflow DEBUG: Task 2002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2002 has memoization hash a1344c2d527f06dfff95724c27d5931c parsl.dataflow.memoization INFO: Task 2002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7707c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2002 try 0 launched on executor multi with executor id 2003 parsl.dataflow.dflow INFO: Standard output for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f4f0c0c3-1803-4dfa-a458-66ca078169fb_isrForCrosstalkSources_1031716_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f4f0c0c3-1803-4dfa-a458-66ca078169fb_isrForCrosstalkSources_1031716_56.stderr parsl.dataflow.dflow DEBUG: Task 2003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2003 has memoization hash 540cdc78063202db29c14ffc88b450cc parsl.dataflow.memoization INFO: Task 2003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2003 try 0 launched on executor multi with executor id 2004 parsl.dataflow.dflow INFO: Standard output for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/2d004444-a75d-4ff3-83e4-608cb3f95a66_isrForCrosstalkSources_1031719_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/2d004444-a75d-4ff3-83e4-608cb3f95a66_isrForCrosstalkSources_1031719_39.stderr parsl.dataflow.dflow DEBUG: Task 2004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2004 has memoization hash 7fe5be2af08701d5ca94c000a0fe5763 parsl.dataflow.memoization INFO: Task 2004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2004 try 0 launched on executor multi with executor id 2005 parsl.dataflow.dflow INFO: Standard output for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8b5d2438-d885-426d-8628-b17fb5cd2682_isrForCrosstalkSources_1031719_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8b5d2438-d885-426d-8628-b17fb5cd2682_isrForCrosstalkSources_1031719_1.stderr parsl.dataflow.dflow DEBUG: Task 2005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2005 has memoization hash fa8143f4e0c795138495199f3d6d4835 parsl.dataflow.memoization INFO: Task 2005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2005 try 0 launched on executor multi with executor id 2006 parsl.dataflow.dflow INFO: Standard output for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/22d0eb95-6f15-442b-b43f-bddd5616f8ae_isrForCrosstalkSources_1031700_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/22d0eb95-6f15-442b-b43f-bddd5616f8ae_isrForCrosstalkSources_1031700_19.stderr parsl.dataflow.dflow DEBUG: Task 2006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2006 has memoization hash 1a468516fafbdca16d88862d60d551a3 parsl.dataflow.memoization INFO: Task 2006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7709a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2006 try 0 launched on executor multi with executor id 2007 parsl.dataflow.dflow INFO: Standard output for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/929b5a66-0f93-4270-a1fb-9bfcb0b6466b_isrForCrosstalkSources_1031710_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/929b5a66-0f93-4270-a1fb-9bfcb0b6466b_isrForCrosstalkSources_1031710_12.stderr parsl.dataflow.dflow DEBUG: Task 2007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2007 has memoization hash b068c5c6d5b58a03d6b4a68eac553278 parsl.dataflow.memoization INFO: Task 2007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2007 try 0 launched on executor multi with executor id 2008 parsl.dataflow.dflow INFO: Standard output for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/87ad27dd-5174-4828-a61d-ca47e231b8ae_isrForCrosstalkSources_1031727_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/87ad27dd-5174-4828-a61d-ca47e231b8ae_isrForCrosstalkSources_1031727_58.stderr parsl.dataflow.dflow DEBUG: Task 2008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2008 has memoization hash ec2ee5c695b8a0e5041bb4d087ebfe75 parsl.dataflow.memoization INFO: Task 2008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2008 try 0 launched on executor multi with executor id 2009 parsl.dataflow.dflow INFO: Standard output for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/ef4a3c28-a471-457f-84df-33c02ed94de0_isrForCrosstalkSources_1031703_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/ef4a3c28-a471-457f-84df-33c02ed94de0_isrForCrosstalkSources_1031703_44.stderr parsl.dataflow.dflow DEBUG: Task 2009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2009 has memoization hash 1f607b822f5ada6738f89a9a3c4756f9 parsl.dataflow.memoization INFO: Task 2009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2009 try 0 launched on executor multi with executor id 2010 parsl.dataflow.dflow INFO: Standard output for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/dd15db76-0d44-4f87-9521-6e270b7b5da2_isrForCrosstalkSources_1031709_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/dd15db76-0d44-4f87-9521-6e270b7b5da2_isrForCrosstalkSources_1031709_43.stderr parsl.dataflow.dflow DEBUG: Task 2010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2010 has memoization hash 7c4a3cc5b7394e5195c27b4e62383199 parsl.dataflow.memoization INFO: Task 2010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2010 try 0 launched on executor multi with executor id 2011 parsl.dataflow.dflow INFO: Standard output for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6c4f137a-d4ff-492f-8361-2031090f1085_isrForCrosstalkSources_1031721_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6c4f137a-d4ff-492f-8361-2031090f1085_isrForCrosstalkSources_1031721_18.stderr parsl.dataflow.dflow DEBUG: Task 2011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2011 has memoization hash 8cd1218013faa46b71267107cb820138 parsl.dataflow.memoization INFO: Task 2011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2011 try 0 launched on executor multi with executor id 2012 parsl.dataflow.dflow INFO: Standard output for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/bc8319a3-0133-4301-9463-64cd89a621e6_isrForCrosstalkSources_1031703_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/bc8319a3-0133-4301-9463-64cd89a621e6_isrForCrosstalkSources_1031703_55.stderr parsl.dataflow.dflow DEBUG: Task 2012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2012 has memoization hash 1abc9ae40232a049a0ff302825c7d8ed parsl.dataflow.memoization INFO: Task 2012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2012 try 0 launched on executor multi with executor id 2013 parsl.dataflow.dflow INFO: Standard output for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/36019044-aea2-444e-be17-cb824a38b7a2_isrForCrosstalkSources_1031707_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/36019044-aea2-444e-be17-cb824a38b7a2_isrForCrosstalkSources_1031707_7.stderr parsl.dataflow.dflow DEBUG: Task 2013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2013 has memoization hash da05617141d0e7d768f5c7715b564e99 parsl.dataflow.memoization INFO: Task 2013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2013 try 0 launched on executor multi with executor id 2014 parsl.dataflow.dflow INFO: Standard output for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/2ffee46c-215a-48b8-853d-7e290f68945b_isrForCrosstalkSources_1031713_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/2ffee46c-215a-48b8-853d-7e290f68945b_isrForCrosstalkSources_1031713_25.stderr parsl.dataflow.dflow DEBUG: Task 2014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2014 has memoization hash 9401afadf5133af71e9fa79b2ee7297d parsl.dataflow.memoization INFO: Task 2014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2014 try 0 launched on executor multi with executor id 2015 parsl.dataflow.dflow INFO: Standard output for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/01f8ce16-779e-4e74-93a8-a3c57409868b_isrForCrosstalkSources_1031722_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/01f8ce16-779e-4e74-93a8-a3c57409868b_isrForCrosstalkSources_1031722_32.stderr parsl.dataflow.dflow DEBUG: Task 2015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2015 has memoization hash 5685da80daca7348611097781638e0b7 parsl.dataflow.memoization INFO: Task 2015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2015 try 0 launched on executor multi with executor id 2016 parsl.dataflow.dflow INFO: Standard output for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/007de0ef-a813-4f36-8041-ba16eb92e07c_isrForCrosstalkSources_1031700_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/007de0ef-a813-4f36-8041-ba16eb92e07c_isrForCrosstalkSources_1031700_47.stderr parsl.dataflow.dflow DEBUG: Task 2016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2016 has memoization hash 589a530426eb53747c27a1de34adf004 parsl.dataflow.memoization INFO: Task 2016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2016 try 0 launched on executor multi with executor id 2017 parsl.dataflow.dflow INFO: Standard output for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ccfdfa4b-c513-4567-8895-28cd961b8ec3_isrForCrosstalkSources_1031726_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ccfdfa4b-c513-4567-8895-28cd961b8ec3_isrForCrosstalkSources_1031726_35.stderr parsl.dataflow.dflow DEBUG: Task 2017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2017 has memoization hash f413af6665e8254feeeb91932c10d37f parsl.dataflow.memoization INFO: Task 2017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2017 try 0 launched on executor multi with executor id 2018 parsl.dataflow.dflow INFO: Standard output for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5e40ecb2-1c39-49b2-82ff-88d697d12c88_isrForCrosstalkSources_1031703_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5e40ecb2-1c39-49b2-82ff-88d697d12c88_isrForCrosstalkSources_1031703_49.stderr parsl.dataflow.dflow DEBUG: Task 2018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2018 has memoization hash c6979d1d5877ad9f30f9f36b3b1ca6ec parsl.dataflow.memoization INFO: Task 2018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2018 try 0 launched on executor multi with executor id 2019 parsl.dataflow.dflow INFO: Standard output for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cd4c79c9-c475-485c-a6a6-e37ee2def71f_isrForCrosstalkSources_1031710_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cd4c79c9-c475-485c-a6a6-e37ee2def71f_isrForCrosstalkSources_1031710_48.stderr parsl.dataflow.dflow DEBUG: Task 2019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2019 has memoization hash b6095326087f7b228370d59bd1cc4d82 parsl.dataflow.memoization INFO: Task 2019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7711c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2019 try 0 launched on executor multi with executor id 2020 parsl.dataflow.dflow INFO: Standard output for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/047a2ece-a10b-4890-ac4e-9e57d77675b3_isrForCrosstalkSources_1031736_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/047a2ece-a10b-4890-ac4e-9e57d77675b3_isrForCrosstalkSources_1031736_49.stderr parsl.dataflow.dflow DEBUG: Task 2020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2020 has memoization hash 061a84fdfd0d8be63dd6281de9377725 parsl.dataflow.memoization INFO: Task 2020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2020 try 0 launched on executor multi with executor id 2021 parsl.dataflow.dflow INFO: Standard output for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/4fd0c0b5-9c80-468b-b28a-a3b03c3bc5c9_isrForCrosstalkSources_1031731_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/4fd0c0b5-9c80-468b-b28a-a3b03c3bc5c9_isrForCrosstalkSources_1031731_27.stderr parsl.dataflow.dflow DEBUG: Task 2021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2021 has memoization hash 956e86f57372d42f29bca3ee010adae6 parsl.dataflow.memoization INFO: Task 2021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2021 try 0 launched on executor multi with executor id 2022 parsl.dataflow.dflow INFO: Standard output for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/8b7a9a36-3c66-42f8-b10f-799e12724254_isrForCrosstalkSources_1031718_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/8b7a9a36-3c66-42f8-b10f-799e12724254_isrForCrosstalkSources_1031718_4.stderr parsl.dataflow.dflow DEBUG: Task 2022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2022 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2022 has memoization hash 0cb04ea4f524bbc23480e098d50b12ba parsl.dataflow.memoization INFO: Task 2022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2022 try 0 launched on executor multi with executor id 2023 parsl.dataflow.dflow INFO: Standard output for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/b038226f-240f-46a7-816b-f67c892824df_isrForCrosstalkSources_1031706_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/b038226f-240f-46a7-816b-f67c892824df_isrForCrosstalkSources_1031706_11.stderr parsl.dataflow.dflow DEBUG: Task 2023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2023 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2023 has memoization hash 5f8a7a89280c375e02415218664fc2a9 parsl.dataflow.memoization INFO: Task 2023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7713a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2023 try 0 launched on executor multi with executor id 2024 parsl.dataflow.dflow INFO: Standard output for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/986953fe-7ed5-43fa-bc15-68ea4568ba4d_isrForCrosstalkSources_1031710_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/986953fe-7ed5-43fa-bc15-68ea4568ba4d_isrForCrosstalkSources_1031710_14.stderr parsl.dataflow.dflow DEBUG: Task 2024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2024 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2024 has memoization hash e95aa48a9b467f64b8b993ebb33e1444 parsl.dataflow.memoization INFO: Task 2024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7714e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2024 try 0 launched on executor multi with executor id 2025 parsl.dataflow.dflow INFO: Standard output for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f5c5580e-d06a-48a4-b0c0-d8e784f7f4f0_isrForCrosstalkSources_1031721_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f5c5580e-d06a-48a4-b0c0-d8e784f7f4f0_isrForCrosstalkSources_1031721_56.stderr parsl.dataflow.dflow DEBUG: Task 2025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2025 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2025 has memoization hash e7bd9a12a05dfd6ec4b580cd47ba63ba parsl.dataflow.memoization INFO: Task 2025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2025 try 0 launched on executor multi with executor id 2026 parsl.dataflow.dflow INFO: Standard output for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/43fd63b3-22e8-452a-b6c7-c6efd6a730ec_isrForCrosstalkSources_1031709_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/43fd63b3-22e8-452a-b6c7-c6efd6a730ec_isrForCrosstalkSources_1031709_11.stderr parsl.dataflow.dflow DEBUG: Task 2026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2026 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2026 has memoization hash 488e16366b13d9e3d559d3bb56bf7bea parsl.dataflow.memoization INFO: Task 2026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2026 try 0 launched on executor multi with executor id 2027 parsl.dataflow.dflow INFO: Standard output for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/87131a1a-56ec-4836-b1db-7baae0e9d87d_isrForCrosstalkSources_1031702_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/87131a1a-56ec-4836-b1db-7baae0e9d87d_isrForCrosstalkSources_1031702_17.stderr parsl.dataflow.dflow DEBUG: Task 2027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2027 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2027 has memoization hash 048e5f6561bcab6969188ae7fab7cd9d parsl.dataflow.memoization INFO: Task 2027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2027 try 0 launched on executor multi with executor id 2028 parsl.dataflow.dflow INFO: Standard output for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/b715adc2-cc09-4fe5-94e4-213097143b00_isrForCrosstalkSources_1031738_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/b715adc2-cc09-4fe5-94e4-213097143b00_isrForCrosstalkSources_1031738_41.stderr parsl.dataflow.dflow DEBUG: Task 2028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2028 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2028 has memoization hash 6066aaba36a4a41ae87b2e049b376d4e parsl.dataflow.memoization INFO: Task 2028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7716c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2028 try 0 launched on executor multi with executor id 2029 parsl.dataflow.dflow INFO: Standard output for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/633aec0c-844e-41e9-aabe-85f43458caf8_isrForCrosstalkSources_1031721_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/633aec0c-844e-41e9-aabe-85f43458caf8_isrForCrosstalkSources_1031721_41.stderr parsl.dataflow.dflow DEBUG: Task 2029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2029 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2029 has memoization hash 170646488cd38d3f8f5cf08031c6b4a3 parsl.dataflow.memoization INFO: Task 2029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2029 try 0 launched on executor multi with executor id 2030 parsl.dataflow.dflow INFO: Standard output for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a5cb0be1-fec7-4f5d-aecb-d946517cff4f_isrForCrosstalkSources_1031702_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a5cb0be1-fec7-4f5d-aecb-d946517cff4f_isrForCrosstalkSources_1031702_31.stderr parsl.dataflow.dflow DEBUG: Task 2030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2030 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2030 has memoization hash 79383474c315f438a20a2057064b2cf2 parsl.dataflow.memoization INFO: Task 2030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7718a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2030 try 0 launched on executor multi with executor id 2031 parsl.dataflow.dflow INFO: Standard output for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/15c78d43-6e16-4a74-8a94-3f7d1053c6bd_isrForCrosstalkSources_1031721_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/15c78d43-6e16-4a74-8a94-3f7d1053c6bd_isrForCrosstalkSources_1031721_27.stderr parsl.dataflow.dflow DEBUG: Task 2031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2031 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2031 has memoization hash 54fa7ce0934262dd6f142fa4ed731733 parsl.dataflow.memoization INFO: Task 2031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2031 try 0 launched on executor multi with executor id 2032 parsl.dataflow.dflow INFO: Standard output for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/eaf76b80-ae3b-4a38-8470-109be1cbc867_isrForCrosstalkSources_1031700_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/eaf76b80-ae3b-4a38-8470-109be1cbc867_isrForCrosstalkSources_1031700_8.stderr parsl.dataflow.dflow DEBUG: Task 2032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2032 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2032 has memoization hash 1f1e4809f0e3be442377fa3e8009f3c1 parsl.dataflow.memoization INFO: Task 2032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2032 try 0 launched on executor multi with executor id 2033 parsl.dataflow.dflow INFO: Standard output for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0e1473cc-5d94-4c17-bfa1-6beadb7219aa_isrForCrosstalkSources_1031720_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0e1473cc-5d94-4c17-bfa1-6beadb7219aa_isrForCrosstalkSources_1031720_33.stderr parsl.dataflow.dflow DEBUG: Task 2033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2033 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2033 has memoization hash ac287a4139d358c98084a2d9f4b4b0c5 parsl.dataflow.memoization INFO: Task 2033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7719e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2033 try 0 launched on executor multi with executor id 2034 parsl.dataflow.dflow INFO: Standard output for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a1836b94-487a-421b-b037-827a89108969_isrForCrosstalkSources_1031738_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a1836b94-487a-421b-b037-827a89108969_isrForCrosstalkSources_1031738_57.stderr parsl.dataflow.dflow DEBUG: Task 2034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2034 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2034 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2034 has memoization hash 33fea7e52817a3635224488cd5315121 parsl.dataflow.memoization INFO: Task 2034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2034 try 0 launched on executor multi with executor id 2035 parsl.dataflow.dflow INFO: Standard output for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/4ce6dbe9-ecd9-4fc2-8230-6ca12b917c56_isrForCrosstalkSources_1031710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/4ce6dbe9-ecd9-4fc2-8230-6ca12b917c56_isrForCrosstalkSources_1031710_2.stderr parsl.dataflow.dflow DEBUG: Task 2035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2035 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2035 has memoization hash 734369f37f0b0f18a09692ebb59bdf02 parsl.dataflow.memoization INFO: Task 2035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2035 try 0 launched on executor multi with executor id 2036 parsl.dataflow.dflow INFO: Standard output for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d4a9a63b-e80c-43b4-b852-43c5b5503ca4_isrForCrosstalkSources_1031714_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d4a9a63b-e80c-43b4-b852-43c5b5503ca4_isrForCrosstalkSources_1031714_49.stderr parsl.dataflow.dflow DEBUG: Task 2036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2036 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2036 has memoization hash 71daa89d8bda89fd30b622f43d5202bd parsl.dataflow.memoization INFO: Task 2036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2036 try 0 launched on executor multi with executor id 2037 parsl.dataflow.dflow INFO: Standard output for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5975855d-a01d-4385-b562-b720038d9e39_isrForCrosstalkSources_1031727_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5975855d-a01d-4385-b562-b720038d9e39_isrForCrosstalkSources_1031727_37.stderr parsl.dataflow.dflow DEBUG: Task 2037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2037 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2037 has memoization hash ec36e0e2cf75b5eb786f6a3535d9f8a5 parsl.dataflow.memoization INFO: Task 2037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2037 try 0 launched on executor multi with executor id 2038 parsl.dataflow.dflow INFO: Standard output for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/bd2fa2cf-334b-44d8-8063-1197ac04538b_isrForCrosstalkSources_1031720_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/bd2fa2cf-334b-44d8-8063-1197ac04538b_isrForCrosstalkSources_1031720_12.stderr parsl.dataflow.dflow DEBUG: Task 2038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2038 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2038 has memoization hash cbcd0ce31d93bf5e2ac68cde59c2f3f5 parsl.dataflow.memoization INFO: Task 2038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2038 try 0 launched on executor multi with executor id 2039 parsl.dataflow.dflow INFO: Standard output for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a06fe411-639a-4a72-9752-f747991a86c8_isrForCrosstalkSources_1031713_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a06fe411-639a-4a72-9752-f747991a86c8_isrForCrosstalkSources_1031713_50.stderr parsl.dataflow.dflow DEBUG: Task 2039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2039 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2039 has memoization hash daa00a87a706c15199ad45f41a1163c7 parsl.dataflow.memoization INFO: Task 2039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2039 try 0 launched on executor multi with executor id 2040 parsl.dataflow.dflow INFO: Standard output for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b1d97f9b-7633-4a04-a2cf-3c2f12228a34_isrForCrosstalkSources_1031703_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b1d97f9b-7633-4a04-a2cf-3c2f12228a34_isrForCrosstalkSources_1031703_6.stderr parsl.dataflow.dflow DEBUG: Task 2040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2040 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2040 has memoization hash e99ec2557d735b162ab8bb79cbf82b09 parsl.dataflow.memoization INFO: Task 2040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2040 try 0 launched on executor multi with executor id 2041 parsl.dataflow.dflow INFO: Standard output for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/06683ed6-ecac-49c2-9cd2-8b3576248c71_isrForCrosstalkSources_1031725_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/06683ed6-ecac-49c2-9cd2-8b3576248c71_isrForCrosstalkSources_1031725_33.stderr parsl.dataflow.dflow DEBUG: Task 2041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2041 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2041 has memoization hash 07b4f2a981b426e041e447a3b989cddc parsl.dataflow.memoization INFO: Task 2041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f771f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2041 try 0 launched on executor multi with executor id 2042 parsl.dataflow.dflow INFO: Standard output for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/9c07404f-0f31-4159-b478-96cacd66af94_isrForCrosstalkSources_1031731_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/9c07404f-0f31-4159-b478-96cacd66af94_isrForCrosstalkSources_1031731_42.stderr parsl.dataflow.dflow DEBUG: Task 2042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2042 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2042 has memoization hash a210e556bb961c431bb7f376cb2b03fc parsl.dataflow.memoization INFO: Task 2042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7720c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2042 try 0 launched on executor multi with executor id 2043 parsl.dataflow.dflow INFO: Standard output for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f343e217-a0a6-4b8d-a4ea-6351b4bf7de3_isrForCrosstalkSources_1031705_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f343e217-a0a6-4b8d-a4ea-6351b4bf7de3_isrForCrosstalkSources_1031705_55.stderr parsl.dataflow.dflow DEBUG: Task 2043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2043 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2043 has memoization hash ea7e086f7f964cae67a4c8ba9661026e parsl.dataflow.memoization INFO: Task 2043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2043 try 0 launched on executor multi with executor id 2044 parsl.dataflow.dflow INFO: Standard output for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/38be037e-c444-47a9-b3e1-bab212e3fd38_isrForCrosstalkSources_1031714_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/38be037e-c444-47a9-b3e1-bab212e3fd38_isrForCrosstalkSources_1031714_2.stderr parsl.dataflow.dflow DEBUG: Task 2044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2044 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2044 has memoization hash f369a0c8233273b626247a598239d43d parsl.dataflow.memoization INFO: Task 2044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2044 try 0 launched on executor multi with executor id 2045 parsl.dataflow.dflow INFO: Standard output for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cc2a89f7-360f-4402-97e2-7ecb98431a4d_isrForCrosstalkSources_1031727_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cc2a89f7-360f-4402-97e2-7ecb98431a4d_isrForCrosstalkSources_1031727_55.stderr parsl.dataflow.dflow DEBUG: Task 2045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2045 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2045 has memoization hash 6c7e7041afc2a66af4b494b22eb576c9 parsl.dataflow.memoization INFO: Task 2045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2045 try 0 launched on executor multi with executor id 2046 parsl.dataflow.dflow INFO: Standard output for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/dd416c02-eba8-4c05-bb83-b755e46afa36_isrForCrosstalkSources_1031728_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/dd416c02-eba8-4c05-bb83-b755e46afa36_isrForCrosstalkSources_1031728_40.stderr parsl.dataflow.dflow DEBUG: Task 2046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2046 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2046 has memoization hash 18b83d566d3568583fdf9fddc95528d4 parsl.dataflow.memoization INFO: Task 2046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7722a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2046 try 0 launched on executor multi with executor id 2047 parsl.dataflow.dflow INFO: Standard output for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/801eeffb-4ff8-4f0d-99ee-bf6ffa3f4211_isrForCrosstalkSources_1031726_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/801eeffb-4ff8-4f0d-99ee-bf6ffa3f4211_isrForCrosstalkSources_1031726_16.stderr parsl.dataflow.dflow DEBUG: Task 2047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2047 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2047 has memoization hash 3ac71f6d6dfda7494651d1ba14b72428 parsl.dataflow.memoization INFO: Task 2047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7723e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2047 try 0 launched on executor multi with executor id 2048 parsl.dataflow.dflow INFO: Standard output for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/affe0b7a-5c7a-4bb0-b4e4-e1d1c193dced_isrForCrosstalkSources_1031727_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/affe0b7a-5c7a-4bb0-b4e4-e1d1c193dced_isrForCrosstalkSources_1031727_24.stderr parsl.dataflow.dflow DEBUG: Task 2048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2048 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2048 has memoization hash e1a7669e39a652a4ae196732ab414fbf parsl.dataflow.memoization INFO: Task 2048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2048 try 0 launched on executor multi with executor id 2049 parsl.dataflow.dflow INFO: Standard output for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/b4dfa5d8-882b-4416-83e6-bfbbb73719bc_isrForCrosstalkSources_1031717_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/b4dfa5d8-882b-4416-83e6-bfbbb73719bc_isrForCrosstalkSources_1031717_22.stderr parsl.dataflow.dflow DEBUG: Task 2049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2049 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2049 has memoization hash a465bfdad9e6747cc14965ce5fbfe4ea parsl.dataflow.memoization INFO: Task 2049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2049 try 0 launched on executor multi with executor id 2050 parsl.dataflow.dflow INFO: Standard output for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/4d12196e-85da-461b-a5e3-a674c45f29b2_isrForCrosstalkSources_1031728_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/4d12196e-85da-461b-a5e3-a674c45f29b2_isrForCrosstalkSources_1031728_17.stderr parsl.dataflow.dflow DEBUG: Task 2050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2050 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2050 has memoization hash 91175689f458be953d25bddabb2af63e parsl.dataflow.memoization INFO: Task 2050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2050 try 0 launched on executor multi with executor id 2051 parsl.dataflow.dflow INFO: Standard output for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7ddf7117-820c-4400-b34d-e1ae6f3d9a7e_isrForCrosstalkSources_1031710_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7ddf7117-820c-4400-b34d-e1ae6f3d9a7e_isrForCrosstalkSources_1031710_52.stderr parsl.dataflow.dflow DEBUG: Task 2051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2051 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2051 has memoization hash 67ad515a0889af877c679dc109808292 parsl.dataflow.memoization INFO: Task 2051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7725c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2051 try 0 launched on executor multi with executor id 2052 parsl.dataflow.dflow INFO: Standard output for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a93108e9-b789-4a8d-8611-2b481e7cc60e_isrForCrosstalkSources_1031720_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a93108e9-b789-4a8d-8611-2b481e7cc60e_isrForCrosstalkSources_1031720_54.stderr parsl.dataflow.dflow DEBUG: Task 2052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2052 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2052 has memoization hash 508287ed185a66317a85c875024b565a parsl.dataflow.memoization INFO: Task 2052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2052 try 0 launched on executor multi with executor id 2053 parsl.dataflow.dflow INFO: Standard output for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/821d87ea-ff40-47a7-9167-d452c56afd49_isrForCrosstalkSources_1031711_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/821d87ea-ff40-47a7-9167-d452c56afd49_isrForCrosstalkSources_1031711_48.stderr parsl.dataflow.dflow DEBUG: Task 2053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2053 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2053 has memoization hash f34658522e04e380998148b990c38247 parsl.dataflow.memoization INFO: Task 2053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2053 try 0 launched on executor multi with executor id 2054 parsl.dataflow.dflow INFO: Standard output for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/6b3a3f22-dde0-4518-a43e-18ea8dc431c8_isrForCrosstalkSources_1031701_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/6b3a3f22-dde0-4518-a43e-18ea8dc431c8_isrForCrosstalkSources_1031701_29.stderr parsl.dataflow.dflow DEBUG: Task 2054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2054 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2054 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2054 has memoization hash e4beef8ae379de2211bb5a7e73e56157 parsl.dataflow.memoization INFO: Task 2054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7727a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2054 try 0 launched on executor multi with executor id 2055 parsl.dataflow.dflow INFO: Standard output for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a2244f1b-2d5f-4770-a277-b116569d1efd_isrForCrosstalkSources_1031712_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a2244f1b-2d5f-4770-a277-b116569d1efd_isrForCrosstalkSources_1031712_30.stderr parsl.dataflow.dflow DEBUG: Task 2055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2055 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2055 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2055 has memoization hash cc2a50ed22e1223ea7f485bca31cea49 parsl.dataflow.memoization INFO: Task 2055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2055 try 0 launched on executor multi with executor id 2056 parsl.dataflow.dflow INFO: Standard output for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/14727c70-ac2f-4aa6-b098-97afcc0f8570_isrForCrosstalkSources_1031727_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/14727c70-ac2f-4aa6-b098-97afcc0f8570_isrForCrosstalkSources_1031727_26.stderr parsl.dataflow.dflow DEBUG: Task 2056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2056 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2056 has memoization hash a616e163f4181e3208dd0e02fda878e8 parsl.dataflow.memoization INFO: Task 2056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7728e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2056 try 0 launched on executor multi with executor id 2057 parsl.dataflow.dflow INFO: Standard output for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/fc821cc4-705e-449f-84d6-8ad84c07b570_isrForCrosstalkSources_1031726_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/fc821cc4-705e-449f-84d6-8ad84c07b570_isrForCrosstalkSources_1031726_5.stderr parsl.dataflow.dflow DEBUG: Task 2057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2057 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2057 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2057 has memoization hash 9940ed408deaaad9083fb9f70bec7491 parsl.dataflow.memoization INFO: Task 2057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2057 try 0 launched on executor multi with executor id 2058 parsl.dataflow.dflow INFO: Standard output for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/fa9a8503-3db9-4224-ba7d-da012d7b094f_isrForCrosstalkSources_1031705_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/fa9a8503-3db9-4224-ba7d-da012d7b094f_isrForCrosstalkSources_1031705_30.stderr parsl.dataflow.dflow DEBUG: Task 2058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2058 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2058 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2058 has memoization hash 9bf47f2e1fbe60ab5066e14c7ace2e6f parsl.dataflow.memoization INFO: Task 2058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2058 try 0 launched on executor multi with executor id 2059 parsl.dataflow.dflow INFO: Standard output for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/d874df06-5068-4f35-ab5e-2c3ce944c0d1_isrForCrosstalkSources_1031716_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/d874df06-5068-4f35-ab5e-2c3ce944c0d1_isrForCrosstalkSources_1031716_62.stderr parsl.dataflow.dflow DEBUG: Task 2059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2059 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2059 has memoization hash 87f82aec12df5571584a0c02164b3261 parsl.dataflow.memoization INFO: Task 2059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2059 try 0 launched on executor multi with executor id 2060 parsl.dataflow.dflow INFO: Standard output for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/d36b3917-a9b5-4b82-9a99-d05f2a599e6c_isrForCrosstalkSources_1031712_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/d36b3917-a9b5-4b82-9a99-d05f2a599e6c_isrForCrosstalkSources_1031712_53.stderr parsl.dataflow.dflow DEBUG: Task 2060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2060 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2060 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2060 has memoization hash 85ffcf2d397c5de9c15c30574de54181 parsl.dataflow.memoization INFO: Task 2060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2060 try 0 launched on executor multi with executor id 2061 parsl.dataflow.dflow INFO: Standard output for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8736a8eb-3bcb-4164-8ac3-f5d3b8796f73_isrForCrosstalkSources_1031703_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8736a8eb-3bcb-4164-8ac3-f5d3b8796f73_isrForCrosstalkSources_1031703_8.stderr parsl.dataflow.dflow DEBUG: Task 2061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2061 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2061 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2061 has memoization hash 963d90e8c14f1900a286d229bf38a5b1 parsl.dataflow.memoization INFO: Task 2061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:21 proc_lsst.multi:170 [INFO] [multi] job_id 17147956 proc_lsst.multi INFO: [multi] job_id 17147956 2024-03-17 04:02:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-17 04:02:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147956 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147956 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17147956 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-17 04:02:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-17 04:02:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-17 04:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2061 try 0 launched on executor multi with executor id 2062 parsl.dataflow.dflow INFO: Standard output for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/0ca97749-eb33-43c4-8a78-b31633281c95_isrForCrosstalkSources_1031733_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/0ca97749-eb33-43c4-8a78-b31633281c95_isrForCrosstalkSources_1031733_8.stderr parsl.dataflow.dflow DEBUG: Task 2062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2062 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2062 has memoization hash 4a3d55067d3905f98604667821eb2c4f parsl.dataflow.memoization INFO: Task 2062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2062 try 0 launched on executor multi with executor id 2063 parsl.dataflow.dflow INFO: Standard output for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/97b3a1de-1ae6-422b-88ee-a96b7d20693c_isrForCrosstalkSources_1031705_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/97b3a1de-1ae6-422b-88ee-a96b7d20693c_isrForCrosstalkSources_1031705_58.stderr parsl.dataflow.dflow DEBUG: Task 2063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2063 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2063 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2063 has memoization hash 6c30ab757c989227e707a1f360755f0a parsl.dataflow.memoization INFO: Task 2063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2063 try 0 launched on executor multi with executor id 2064 parsl.dataflow.dflow INFO: Standard output for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c826ecba-0bd6-45b0-90b8-18963f6f8d09_isrForCrosstalkSources_1031708_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c826ecba-0bd6-45b0-90b8-18963f6f8d09_isrForCrosstalkSources_1031708_40.stderr parsl.dataflow.dflow DEBUG: Task 2064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2064 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2064 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2064 has memoization hash 52772f1e68890ec6fe11e645551decd2 parsl.dataflow.memoization INFO: Task 2064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f9defc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2064 try 0 launched on executor multi with executor id 2065 parsl.dataflow.dflow INFO: Standard output for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c7cf93bf-b887-432f-9b93-c1ec2e680113_isrForCrosstalkSources_1031710_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c7cf93bf-b887-432f-9b93-c1ec2e680113_isrForCrosstalkSources_1031710_7.stderr parsl.dataflow.dflow DEBUG: Task 2065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2065 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2065 has memoization hash e705e2169a9af089dcf941fcc574054e parsl.dataflow.memoization INFO: Task 2065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2065 try 0 launched on executor multi with executor id 2066 parsl.dataflow.dflow INFO: Standard output for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e9bd9524-1fd6-425b-8d28-b182bcada7ad_isrForCrosstalkSources_1031701_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e9bd9524-1fd6-425b-8d28-b182bcada7ad_isrForCrosstalkSources_1031701_6.stderr parsl.dataflow.dflow DEBUG: Task 2066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2066 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2066 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2066 has memoization hash 83acf702e8793e0901c0ca1d971c6983 parsl.dataflow.memoization INFO: Task 2066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2066 try 0 launched on executor multi with executor id 2067 parsl.dataflow.dflow INFO: Standard output for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/c6f0adc0-3a6a-4232-b6fd-8920fa6428fd_isrForCrosstalkSources_1031733_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/c6f0adc0-3a6a-4232-b6fd-8920fa6428fd_isrForCrosstalkSources_1031733_16.stderr parsl.dataflow.dflow DEBUG: Task 2067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2067 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2067 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2067 has memoization hash beae7ab5216c518e000f74c3cb461881 parsl.dataflow.memoization INFO: Task 2067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2067 try 0 launched on executor multi with executor id 2068 parsl.dataflow.dflow INFO: Standard output for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/354f9fe3-481a-470b-9c19-e3bf3e848259_isrForCrosstalkSources_1031699_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/354f9fe3-481a-470b-9c19-e3bf3e848259_isrForCrosstalkSources_1031699_52.stderr parsl.dataflow.dflow DEBUG: Task 2068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2068 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2068 has memoization hash 1fe6c8a4241deb1983151e230597f507 parsl.dataflow.memoization INFO: Task 2068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2068 try 0 launched on executor multi with executor id 2069 parsl.dataflow.dflow INFO: Standard output for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/fe2ed599-7e1f-400d-95d1-3c22a033da19_isrForCrosstalkSources_1031709_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/fe2ed599-7e1f-400d-95d1-3c22a033da19_isrForCrosstalkSources_1031709_32.stderr parsl.dataflow.dflow DEBUG: Task 2069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2069 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2069 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2069 has memoization hash 62f9713da48a2664fec4121ce4505991 parsl.dataflow.memoization INFO: Task 2069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2069 try 0 launched on executor multi with executor id 2070 parsl.dataflow.dflow INFO: Standard output for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/a728f267-1b74-431b-8d62-8af9c469a1c1_isrForCrosstalkSources_1031703_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/a728f267-1b74-431b-8d62-8af9c469a1c1_isrForCrosstalkSources_1031703_57.stderr parsl.dataflow.dflow DEBUG: Task 2070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2070 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2070 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2070 has memoization hash 0283751dd067714236a1a053c8abd36e parsl.dataflow.memoization INFO: Task 2070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7731a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2070 try 0 launched on executor multi with executor id 2071 parsl.dataflow.dflow INFO: Standard output for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7802e1a2-0da4-4524-8e02-3eb52f3dd189_isrForCrosstalkSources_1031710_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7802e1a2-0da4-4524-8e02-3eb52f3dd189_isrForCrosstalkSources_1031710_24.stderr parsl.dataflow.dflow DEBUG: Task 2071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2071 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2071 has memoization hash dfddf1c180c9200e91001ed7ff2ed4fe parsl.dataflow.memoization INFO: Task 2071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2071 try 0 launched on executor multi with executor id 2072 parsl.dataflow.dflow INFO: Standard output for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0850cce1-723b-40b8-b9ac-90c7c620f512_isrForCrosstalkSources_1031721_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0850cce1-723b-40b8-b9ac-90c7c620f512_isrForCrosstalkSources_1031721_20.stderr parsl.dataflow.dflow DEBUG: Task 2072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2072 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2072 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2072 has memoization hash 915a673067f90379957523c3a2c1cc0e parsl.dataflow.memoization INFO: Task 2072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7732e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2072 try 0 launched on executor multi with executor id 2073 parsl.dataflow.dflow INFO: Standard output for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/82d95df7-607f-46a3-894e-7b4a6929259c_isrForCrosstalkSources_1031733_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/82d95df7-607f-46a3-894e-7b4a6929259c_isrForCrosstalkSources_1031733_57.stderr parsl.dataflow.dflow DEBUG: Task 2073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2073 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2073 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2073 has memoization hash 68ccae5a917c80112effcb519129524e parsl.dataflow.memoization INFO: Task 2073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7734c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2073 try 0 launched on executor multi with executor id 2074 parsl.dataflow.dflow INFO: Standard output for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5c3efb72-ac68-447c-b2f5-e3e9e8aa96e4_isrForCrosstalkSources_1031724_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5c3efb72-ac68-447c-b2f5-e3e9e8aa96e4_isrForCrosstalkSources_1031724_23.stderr parsl.dataflow.dflow DEBUG: Task 2074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2074 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2074 has memoization hash d9b4cc234a26bac049cb64699e3754cb parsl.dataflow.memoization INFO: Task 2074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2074 try 0 launched on executor multi with executor id 2075 parsl.dataflow.dflow INFO: Standard output for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/c43bdcae-efb6-483b-8c27-8c5c620f38d6_isrForCrosstalkSources_1031714_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/c43bdcae-efb6-483b-8c27-8c5c620f38d6_isrForCrosstalkSources_1031714_25.stderr parsl.dataflow.dflow DEBUG: Task 2075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2075 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2075 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2075 has memoization hash 6467d422d845fddef67e279e2e4d8963 parsl.dataflow.memoization INFO: Task 2075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2075 try 0 launched on executor multi with executor id 2076 parsl.dataflow.dflow INFO: Standard output for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/75189c41-6a89-41c6-b86a-5a543971cc94_isrForCrosstalkSources_1031738_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/75189c41-6a89-41c6-b86a-5a543971cc94_isrForCrosstalkSources_1031738_37.stderr parsl.dataflow.dflow DEBUG: Task 2076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2076 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2076 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2076 has memoization hash d76bc6e52c8debb1c880990629f72a4f parsl.dataflow.memoization INFO: Task 2076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2076 try 0 launched on executor multi with executor id 2077 parsl.dataflow.dflow INFO: Standard output for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/9bc3bbea-3f20-49c4-a521-925f831c4ca4_isrForCrosstalkSources_1031708_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/9bc3bbea-3f20-49c4-a521-925f831c4ca4_isrForCrosstalkSources_1031708_51.stderr parsl.dataflow.dflow DEBUG: Task 2077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2077 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2077 has memoization hash 1f203fce52bdbb04808202f738845e62 parsl.dataflow.memoization INFO: Task 2077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2077 try 0 launched on executor multi with executor id 2078 parsl.dataflow.dflow INFO: Standard output for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0dc81d4d-3509-4279-b2e1-c48647d86725_isrForCrosstalkSources_1031737_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0dc81d4d-3509-4279-b2e1-c48647d86725_isrForCrosstalkSources_1031737_3.stderr parsl.dataflow.dflow DEBUG: Task 2078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2078 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2078 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2078 has memoization hash 0027aa9de1153a1d90f542d54e05296d parsl.dataflow.memoization INFO: Task 2078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2078 try 0 launched on executor multi with executor id 2079 parsl.dataflow.dflow INFO: Standard output for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/a2205d95-88d3-4147-8454-95fd1708d097_isrForCrosstalkSources_1031714_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/a2205d95-88d3-4147-8454-95fd1708d097_isrForCrosstalkSources_1031714_60.stderr parsl.dataflow.dflow DEBUG: Task 2079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2079 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2079 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2079 has memoization hash 255a39d497b4b90026d5124842414888 parsl.dataflow.memoization INFO: Task 2079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7736a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2079 try 0 launched on executor multi with executor id 2080 parsl.dataflow.dflow INFO: Standard output for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/f0fb0dc2-dab6-46a8-a3cf-5fdcc4b2c588_isrForCrosstalkSources_1031701_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/f0fb0dc2-dab6-46a8-a3cf-5fdcc4b2c588_isrForCrosstalkSources_1031701_30.stderr parsl.dataflow.dflow DEBUG: Task 2080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2080 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2080 has memoization hash a1b290b46f6c124f7a1cd2ff7bdc9920 parsl.dataflow.memoization INFO: Task 2080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2080 try 0 launched on executor multi with executor id 2081 parsl.dataflow.dflow INFO: Standard output for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7e6f518a-011e-4a70-8ded-72bc0c6a4767_isrForCrosstalkSources_1031701_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7e6f518a-011e-4a70-8ded-72bc0c6a4767_isrForCrosstalkSources_1031701_2.stderr parsl.dataflow.dflow DEBUG: Task 2081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2081 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2081 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2081 has memoization hash a4b1214a3f3dbc5539cd36c265566c9b parsl.dataflow.memoization INFO: Task 2081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2081 try 0 launched on executor multi with executor id 2082 parsl.dataflow.dflow INFO: Standard output for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4f5a1ca9-6cbe-4884-b610-430ae5f3f6d6_isrForCrosstalkSources_1031720_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4f5a1ca9-6cbe-4884-b610-430ae5f3f6d6_isrForCrosstalkSources_1031720_1.stderr parsl.dataflow.dflow DEBUG: Task 2082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2082 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2082 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2082 has memoization hash 73afe6450eceb8d149dbed63e6879103 parsl.dataflow.memoization INFO: Task 2082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7739c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2082 try 0 launched on executor multi with executor id 2083 parsl.dataflow.dflow INFO: Standard output for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/545e0216-6e76-419a-a86c-711751d92e27_isrForCrosstalkSources_1031709_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/545e0216-6e76-419a-a86c-711751d92e27_isrForCrosstalkSources_1031709_20.stderr parsl.dataflow.dflow DEBUG: Task 2083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2083 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2083 has memoization hash 17cbf04c20eb8b439b19434a8dbbd7e3 parsl.dataflow.memoization INFO: Task 2083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2083 try 0 launched on executor multi with executor id 2084 parsl.dataflow.dflow INFO: Standard output for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/b9986e23-e533-43fc-87ff-04ff20b93b0c_isrForCrosstalkSources_1031736_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/b9986e23-e533-43fc-87ff-04ff20b93b0c_isrForCrosstalkSources_1031736_6.stderr parsl.dataflow.dflow DEBUG: Task 2084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2084 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2084 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2084 has memoization hash 228d760c7f98f0b05d0c73ea3a2f3bf7 parsl.dataflow.memoization INFO: Task 2084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f7737e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2084 try 0 launched on executor multi with executor id 2085 parsl.dataflow.dflow INFO: Standard output for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f775baee-7f95-4515-8146-a2fad114f238_isrForCrosstalkSources_1031706_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f775baee-7f95-4515-8146-a2fad114f238_isrForCrosstalkSources_1031706_1.stderr parsl.dataflow.dflow DEBUG: Task 2085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2085 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2085 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2085 has memoization hash e03e1eae991194145dae2bd9802b62a1 parsl.dataflow.memoization INFO: Task 2085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2085 try 0 launched on executor multi with executor id 2086 parsl.dataflow.dflow INFO: Standard output for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/eaf6898c-261d-4d11-a16d-18bd2024b25a_isrForCrosstalkSources_1031729_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/eaf6898c-261d-4d11-a16d-18bd2024b25a_isrForCrosstalkSources_1031729_59.stderr parsl.dataflow.dflow DEBUG: Task 2086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2086 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2086 has memoization hash 5cb78cd77ee26dcbd7510bb10a7d2852 parsl.dataflow.memoization INFO: Task 2086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2086 try 0 launched on executor multi with executor id 2087 parsl.dataflow.dflow INFO: Standard output for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/d0fdf29e-38d0-49fa-8d69-e5fc42f90b1b_isrForCrosstalkSources_1031702_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/d0fdf29e-38d0-49fa-8d69-e5fc42f90b1b_isrForCrosstalkSources_1031702_42.stderr parsl.dataflow.dflow DEBUG: Task 2087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2087 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2087 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2087 has memoization hash e06f8182c432425d4e79538bceaef916 parsl.dataflow.memoization INFO: Task 2087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2087 try 0 launched on executor multi with executor id 2088 parsl.dataflow.dflow INFO: Standard output for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/6884f24a-93a4-4694-9929-3cb5d1e0a667_isrForCrosstalkSources_1031729_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/6884f24a-93a4-4694-9929-3cb5d1e0a667_isrForCrosstalkSources_1031729_2.stderr parsl.dataflow.dflow DEBUG: Task 2088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2088 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2088 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2088 has memoization hash 4a5d929a2bada9daae26ed56bc01ec90 parsl.dataflow.memoization INFO: Task 2088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2088 try 0 launched on executor multi with executor id 2089 parsl.dataflow.dflow INFO: Standard output for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e33b2dc4-327e-4b9b-b542-b56c3edf1e10_isrForCrosstalkSources_1031701_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e33b2dc4-327e-4b9b-b542-b56c3edf1e10_isrForCrosstalkSources_1031701_52.stderr parsl.dataflow.dflow DEBUG: Task 2089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2089 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2089 has memoization hash ee92385d77cd433f5ee15c3b6d1eef2f parsl.dataflow.memoization INFO: Task 2089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2089 try 0 launched on executor multi with executor id 2090 parsl.dataflow.dflow INFO: Standard output for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/fd22232f-4b7b-4e2f-add7-2c2106f5496a_isrForCrosstalkSources_1031722_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/fd22232f-4b7b-4e2f-add7-2c2106f5496a_isrForCrosstalkSources_1031722_23.stderr parsl.dataflow.dflow DEBUG: Task 2090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2090 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2090 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2090 has memoization hash 5741087635a3661929c8613f039cbf71 parsl.dataflow.memoization INFO: Task 2090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2090 try 0 launched on executor multi with executor id 2091 parsl.dataflow.dflow INFO: Standard output for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/be0eb3e7-dc31-4629-85e8-0432a38dc9b0_isrForCrosstalkSources_1031713_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/be0eb3e7-dc31-4629-85e8-0432a38dc9b0_isrForCrosstalkSources_1031713_10.stderr parsl.dataflow.dflow DEBUG: Task 2091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2091 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2091 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2091 has memoization hash 0d89b8ba6d9206fddbbd053359fb75ef parsl.dataflow.memoization INFO: Task 2091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2091 try 0 launched on executor multi with executor id 2092 parsl.dataflow.dflow INFO: Standard output for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ccd19f9f-3bc4-4c13-8a57-2628a227ba5f_isrForCrosstalkSources_1031724_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ccd19f9f-3bc4-4c13-8a57-2628a227ba5f_isrForCrosstalkSources_1031724_59.stderr parsl.dataflow.dflow DEBUG: Task 2092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2092 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2092 has memoization hash 599508eb53a19a5547a9b1046c3c62bb parsl.dataflow.memoization INFO: Task 2092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2092 try 0 launched on executor multi with executor id 2093 parsl.dataflow.dflow INFO: Standard output for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6eb8f699-1b6c-47c0-ba01-0bba9b0fdd16_isrForCrosstalkSources_1031700_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6eb8f699-1b6c-47c0-ba01-0bba9b0fdd16_isrForCrosstalkSources_1031700_56.stderr parsl.dataflow.dflow DEBUG: Task 2093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2093 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2093 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2093 has memoization hash cfcaea23af2f410e3cc7effdf67ede5f parsl.dataflow.memoization INFO: Task 2093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f770040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2093 try 0 launched on executor multi with executor id 2094 parsl.dataflow.dflow INFO: Standard output for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/18e31040-61f0-46b3-be42-ed19bc548593_isrForCrosstalkSources_1031735_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/18e31040-61f0-46b3-be42-ed19bc548593_isrForCrosstalkSources_1031735_30.stderr parsl.dataflow.dflow DEBUG: Task 2094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2094 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2094 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2094 has memoization hash 5324edacc196167f1a1ab64a2098afd2 parsl.dataflow.memoization INFO: Task 2094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6040e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2094 try 0 launched on executor multi with executor id 2095 parsl.dataflow.dflow INFO: Standard output for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8f51add8-8787-4bc2-83d0-7da1c1dc1d02_isrForCrosstalkSources_1031730_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8f51add8-8787-4bc2-83d0-7da1c1dc1d02_isrForCrosstalkSources_1031730_9.stderr parsl.dataflow.dflow DEBUG: Task 2095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2095 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2095 has memoization hash 9f165bd7143d6b19677a635bb46fe830 parsl.dataflow.memoization INFO: Task 2095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2095 try 0 launched on executor multi with executor id 2096 parsl.dataflow.dflow INFO: Standard output for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/a93a6028-9d83-46e3-9557-4440169da3f5_isrForCrosstalkSources_1031715_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/a93a6028-9d83-46e3-9557-4440169da3f5_isrForCrosstalkSources_1031715_35.stderr parsl.dataflow.dflow DEBUG: Task 2096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2096 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2096 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2096 has memoization hash c85acd8f2221cb5f44e7a80478f489fc parsl.dataflow.memoization INFO: Task 2096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2096 try 0 launched on executor multi with executor id 2097 parsl.dataflow.dflow INFO: Standard output for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/2e2ae47e-5ee8-4bd9-b8a2-333f7d307c69_isrForCrosstalkSources_1031725_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/2e2ae47e-5ee8-4bd9-b8a2-333f7d307c69_isrForCrosstalkSources_1031725_23.stderr parsl.dataflow.dflow DEBUG: Task 2097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2097 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2097 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2097 has memoization hash 8b910d6d0faf4d33b876d7edb9836a10 parsl.dataflow.memoization INFO: Task 2097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2097 try 0 launched on executor multi with executor id 2098 parsl.dataflow.dflow INFO: Standard output for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/10a6ce14-3a29-4b02-b249-0e5e4e90c32b_isrForCrosstalkSources_1031711_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/10a6ce14-3a29-4b02-b249-0e5e4e90c32b_isrForCrosstalkSources_1031711_19.stderr parsl.dataflow.dflow DEBUG: Task 2098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2098 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2098 has memoization hash 69edd51cab1a0cc1cb5cc6acffca5475 parsl.dataflow.memoization INFO: Task 2098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6042c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2098 try 0 launched on executor multi with executor id 2099 parsl.dataflow.dflow INFO: Standard output for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/eff85d48-05e5-498c-9e8c-754a4e52eea6_isrForCrosstalkSources_1031699_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/eff85d48-05e5-498c-9e8c-754a4e52eea6_isrForCrosstalkSources_1031699_34.stderr parsl.dataflow.dflow DEBUG: Task 2099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2099 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2099 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2099 has memoization hash 3ff4c2da81639dd95c920a9d0b2d183f parsl.dataflow.memoization INFO: Task 2099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2099 try 0 launched on executor multi with executor id 2100 parsl.dataflow.dflow INFO: Standard output for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/83325b05-ce75-41a0-868d-93b94a2e2e22_isrForCrosstalkSources_1031725_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/83325b05-ce75-41a0-868d-93b94a2e2e22_isrForCrosstalkSources_1031725_32.stderr parsl.dataflow.dflow DEBUG: Task 2100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2100 has memoization hash bff5f791a0fa67f5b9bd247f9e711d9e parsl.dataflow.memoization INFO: Task 2100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6044a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2100 try 0 launched on executor multi with executor id 2101 parsl.dataflow.dflow INFO: Standard output for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/00d562b3-44dd-432b-8552-9be5981a01f0_isrForCrosstalkSources_1031710_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/00d562b3-44dd-432b-8552-9be5981a01f0_isrForCrosstalkSources_1031710_35.stderr parsl.dataflow.dflow DEBUG: Task 2101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2101 has memoization hash 2d22d76195aaabb1b7054ea8123933c7 parsl.dataflow.memoization INFO: Task 2101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2101 try 0 launched on executor multi with executor id 2102 parsl.dataflow.dflow INFO: Standard output for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/04af23cd-7287-49b5-9f86-670da5ededb1_isrForCrosstalkSources_1031720_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/04af23cd-7287-49b5-9f86-670da5ededb1_isrForCrosstalkSources_1031720_28.stderr parsl.dataflow.dflow DEBUG: Task 2102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2102 has memoization hash be46127ff286ff65f69dab747035b715 parsl.dataflow.memoization INFO: Task 2102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2102 try 0 launched on executor multi with executor id 2103 parsl.dataflow.dflow INFO: Standard output for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/0ab90373-c1ef-4748-b96a-408c88e94118_isrForCrosstalkSources_1031730_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/0ab90373-c1ef-4748-b96a-408c88e94118_isrForCrosstalkSources_1031730_62.stderr parsl.dataflow.dflow DEBUG: Task 2103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2103 has memoization hash eabcbffe9bb0d07e394e5b798e55a415 parsl.dataflow.memoization INFO: Task 2103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6045e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2103 try 0 launched on executor multi with executor id 2104 parsl.dataflow.dflow INFO: Standard output for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/271966b6-8983-41e6-b306-ee0cef328c23_isrForCrosstalkSources_1031706_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/271966b6-8983-41e6-b306-ee0cef328c23_isrForCrosstalkSources_1031706_39.stderr parsl.dataflow.dflow DEBUG: Task 2104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2104 has memoization hash fc218ac2bf1f1767b5473162b3a1e258 parsl.dataflow.memoization INFO: Task 2104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2104 try 0 launched on executor multi with executor id 2105 parsl.dataflow.dflow INFO: Standard output for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c30ba4a2-8df0-4aa5-beed-233bb1b64608_isrForCrosstalkSources_1031707_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c30ba4a2-8df0-4aa5-beed-233bb1b64608_isrForCrosstalkSources_1031707_57.stderr parsl.dataflow.dflow DEBUG: Task 2105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2105 has memoization hash c9fde32fdfc036e761bc87bb6d6d020a parsl.dataflow.memoization INFO: Task 2105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6047c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2105 try 0 launched on executor multi with executor id 2106 parsl.dataflow.dflow INFO: Standard output for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7b24ac51-30f5-4f32-978b-1cfb57d181b5_isrForCrosstalkSources_1031701_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7b24ac51-30f5-4f32-978b-1cfb57d181b5_isrForCrosstalkSources_1031701_21.stderr parsl.dataflow.dflow DEBUG: Task 2106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2106 has memoization hash af47e27e50065b4c55a9c9b371c24c3a parsl.dataflow.memoization INFO: Task 2106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2106 try 0 launched on executor multi with executor id 2107 parsl.dataflow.dflow INFO: Standard output for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/98d17e8e-a505-44b1-87c9-7686978555ba_isrForCrosstalkSources_1031713_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/98d17e8e-a505-44b1-87c9-7686978555ba_isrForCrosstalkSources_1031713_3.stderr parsl.dataflow.dflow DEBUG: Task 2107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2107 has memoization hash e820c2240c0e2cdfb23b360cc84f892b parsl.dataflow.memoization INFO: Task 2107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2107 try 0 launched on executor multi with executor id 2108 parsl.dataflow.dflow INFO: Standard output for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/0512db29-23db-42ea-a79c-99cde9de641d_isrForCrosstalkSources_1031729_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/0512db29-23db-42ea-a79c-99cde9de641d_isrForCrosstalkSources_1031729_43.stderr parsl.dataflow.dflow DEBUG: Task 2108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2108 has memoization hash d1c62777449262bf1d92c262ea4ff47b parsl.dataflow.memoization INFO: Task 2108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2108 try 0 launched on executor multi with executor id 2109 parsl.dataflow.dflow INFO: Standard output for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d42b19ae-37a3-46f9-b80a-465b39198371_isrForCrosstalkSources_1031733_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d42b19ae-37a3-46f9-b80a-465b39198371_isrForCrosstalkSources_1031733_49.stderr parsl.dataflow.dflow DEBUG: Task 2109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2109 has memoization hash b64ecd92619bca12c69d119401544f38 parsl.dataflow.memoization INFO: Task 2109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2109 try 0 launched on executor multi with executor id 2110 parsl.dataflow.dflow INFO: Standard output for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2d222ea4-973e-4b3f-9895-4061edec616b_isrForCrosstalkSources_1031724_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2d222ea4-973e-4b3f-9895-4061edec616b_isrForCrosstalkSources_1031724_49.stderr parsl.dataflow.dflow DEBUG: Task 2110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2110 has memoization hash ab62aa00b14247934ba6963b3cf4f7d5 parsl.dataflow.memoization INFO: Task 2110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2110 try 0 launched on executor multi with executor id 2111 parsl.dataflow.dflow INFO: Standard output for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/7050e05c-adaf-47cb-a8d7-b35edfe25cff_isrForCrosstalkSources_1031718_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/7050e05c-adaf-47cb-a8d7-b35edfe25cff_isrForCrosstalkSources_1031718_46.stderr parsl.dataflow.dflow DEBUG: Task 2111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2111 has memoization hash 1b9bd052495a671c0846f88ca93b0a6c parsl.dataflow.memoization INFO: Task 2111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2111 try 0 launched on executor multi with executor id 2112 parsl.dataflow.dflow INFO: Standard output for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3b70ad21-16c8-4382-80b4-a4556b7a5f5a_isrForCrosstalkSources_1031736_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3b70ad21-16c8-4382-80b4-a4556b7a5f5a_isrForCrosstalkSources_1031736_8.stderr parsl.dataflow.dflow DEBUG: Task 2112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2112 has memoization hash e6e7c1386e05e8a317d931821404e2ac parsl.dataflow.memoization INFO: Task 2112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2112 try 0 launched on executor multi with executor id 2113 parsl.dataflow.dflow INFO: Standard output for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c4313041-7ee8-49d2-b1cb-07d3a8756cb7_isrForCrosstalkSources_1031731_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c4313041-7ee8-49d2-b1cb-07d3a8756cb7_isrForCrosstalkSources_1031731_29.stderr parsl.dataflow.dflow DEBUG: Task 2113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2113 has memoization hash 91a371de952db301585fca17d7b3ceaa parsl.dataflow.memoization INFO: Task 2113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2113 try 0 launched on executor multi with executor id 2114 parsl.dataflow.dflow INFO: Standard output for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6da3e724-0cbd-4fc4-b9ad-71e5f5c19d41_isrForCrosstalkSources_1031723_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6da3e724-0cbd-4fc4-b9ad-71e5f5c19d41_isrForCrosstalkSources_1031723_35.stderr parsl.dataflow.dflow DEBUG: Task 2114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2114 has memoization hash 1c4de516cc822cac9b6f1d93c9767b8d parsl.dataflow.memoization INFO: Task 2114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2114 try 0 launched on executor multi with executor id 2115 parsl.dataflow.dflow INFO: Standard output for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/d9b02641-8c01-4dc3-922c-d02ebda04d37_isrForCrosstalkSources_1031717_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/d9b02641-8c01-4dc3-922c-d02ebda04d37_isrForCrosstalkSources_1031717_18.stderr parsl.dataflow.dflow DEBUG: Task 2115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2115 has memoization hash 3a01572a3921245826863bd8bd6d9b05 parsl.dataflow.memoization INFO: Task 2115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2115 try 0 launched on executor multi with executor id 2116 parsl.dataflow.dflow INFO: Standard output for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/66462ffe-5330-45fa-b744-f679be70eb9c_isrForCrosstalkSources_1031729_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/66462ffe-5330-45fa-b744-f679be70eb9c_isrForCrosstalkSources_1031729_14.stderr parsl.dataflow.dflow DEBUG: Task 2116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2116 has memoization hash ba4148d2fff30c90c6856a1a4e6ac760 parsl.dataflow.memoization INFO: Task 2116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2116 try 0 launched on executor multi with executor id 2117 parsl.dataflow.dflow INFO: Standard output for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5cd3bd48-443b-4ffa-93ab-6eb9a8c1dbad_isrForCrosstalkSources_1031728_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5cd3bd48-443b-4ffa-93ab-6eb9a8c1dbad_isrForCrosstalkSources_1031728_43.stderr parsl.dataflow.dflow DEBUG: Task 2117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2117 has memoization hash 684f3dcbc2a1ebf844b57d6305461d27 parsl.dataflow.memoization INFO: Task 2117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2117 try 0 launched on executor multi with executor id 2118 parsl.dataflow.dflow INFO: Standard output for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/d6d5df29-b354-48c6-ad76-ce783640552d_isrForCrosstalkSources_1031721_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/d6d5df29-b354-48c6-ad76-ce783640552d_isrForCrosstalkSources_1031721_21.stderr parsl.dataflow.dflow DEBUG: Task 2118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2118 has memoization hash bdf19b4f511ba636270d05a3a7573f89 parsl.dataflow.memoization INFO: Task 2118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2118 try 0 launched on executor multi with executor id 2119 parsl.dataflow.dflow INFO: Standard output for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67c07326-bed6-46ee-952a-fd9764e1040c_isrForCrosstalkSources_1031728_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67c07326-bed6-46ee-952a-fd9764e1040c_isrForCrosstalkSources_1031728_22.stderr parsl.dataflow.dflow DEBUG: Task 2119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2119 has memoization hash 9855ce9e9f103aee7ba4fc5fc615c460 parsl.dataflow.memoization INFO: Task 2119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2119 try 0 launched on executor multi with executor id 2120 parsl.dataflow.dflow INFO: Standard output for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/1cf7ab46-847a-4a63-82b8-2dc42d55a87b_isrForCrosstalkSources_1031717_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/1cf7ab46-847a-4a63-82b8-2dc42d55a87b_isrForCrosstalkSources_1031717_15.stderr parsl.dataflow.dflow DEBUG: Task 2120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2120 has memoization hash d94de5745de5059cc2b19e81d9a705c8 parsl.dataflow.memoization INFO: Task 2120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2120 try 0 launched on executor multi with executor id 2121 parsl.dataflow.dflow INFO: Standard output for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/094b63be-3559-450f-925a-03f891a67324_isrForCrosstalkSources_1031724_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/094b63be-3559-450f-925a-03f891a67324_isrForCrosstalkSources_1031724_10.stderr parsl.dataflow.dflow DEBUG: Task 2121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2121 has memoization hash ad202af6a755d91bcfd23f4b31469cae parsl.dataflow.memoization INFO: Task 2121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6051c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2121 try 0 launched on executor multi with executor id 2122 parsl.dataflow.dflow INFO: Standard output for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/3e96e247-5579-4c8d-9401-5d514391972c_isrForCrosstalkSources_1031717_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/3e96e247-5579-4c8d-9401-5d514391972c_isrForCrosstalkSources_1031717_48.stderr parsl.dataflow.dflow DEBUG: Task 2122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2122 has memoization hash 7f58fd129526fd3f6443b312b9b5cfe1 parsl.dataflow.memoization INFO: Task 2122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2122 try 0 launched on executor multi with executor id 2123 parsl.dataflow.dflow INFO: Standard output for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ad5baa43-af15-4da0-8233-12d9e71dee7c_isrForCrosstalkSources_1031713_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ad5baa43-af15-4da0-8233-12d9e71dee7c_isrForCrosstalkSources_1031713_42.stderr parsl.dataflow.dflow DEBUG: Task 2123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2123 has memoization hash 5b79406cd25f8cb2a378ce4242b68237 parsl.dataflow.memoization INFO: Task 2123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2123 try 0 launched on executor multi with executor id 2124 parsl.dataflow.dflow INFO: Standard output for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/790ad14b-b5d0-4678-8f63-d0e94c7c27ca_isrForCrosstalkSources_1031733_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/790ad14b-b5d0-4678-8f63-d0e94c7c27ca_isrForCrosstalkSources_1031733_20.stderr parsl.dataflow.dflow DEBUG: Task 2124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2124 has memoization hash 5a51d27cda18bab14c91c8de8b938ea9 parsl.dataflow.memoization INFO: Task 2124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6053a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fa04819b975c92afb377c1cfad97545 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/54691db5-5817-4ea4-9c1f-4cf029786ae5_isrForCrosstalkSources_1031721_61.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/54691db5-5817-4ea4-9c1f-4cf029786ae5_isrForCrosstalkSources_1031721_61.stderr parsl.dataflow.dflow INFO: Parsl task 2124 try 0 launched on executor multi with executor id 2125 parsl.dataflow.dflow INFO: Standard output for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/b1d64d71-fcaa-451f-a8c0-b42bab019285_isrForCrosstalkSources_1031709_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/b1d64d71-fcaa-451f-a8c0-b42bab019285_isrForCrosstalkSources_1031709_45.stderr parsl.dataflow.dflow DEBUG: Task 2125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2125 has memoization hash d973d7a2a7f980d5d409f59989a981b9 parsl.dataflow.memoization INFO: Task 2125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2125 try 0 launched on executor multi with executor id 2126 parsl.dataflow.dflow INFO: Standard output for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/58aaf35f-3343-4413-8cbd-c8a7553e5afd_isrForCrosstalkSources_1031725_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/58aaf35f-3343-4413-8cbd-c8a7553e5afd_isrForCrosstalkSources_1031725_57.stderr parsl.dataflow.dflow DEBUG: Task 2126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2126 has memoization hash 4f6268eeb50048aad86794af733cb846 parsl.dataflow.memoization INFO: Task 2126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2126 try 0 launched on executor multi with executor id 2127 parsl.dataflow.dflow INFO: Standard output for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/404bca67-edb1-425e-8914-062e0c9ba8aa_isrForCrosstalkSources_1031725_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/404bca67-edb1-425e-8914-062e0c9ba8aa_isrForCrosstalkSources_1031725_12.stderr parsl.dataflow.dflow DEBUG: Task 2127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2127 has memoization hash 899ac16c355963586ad5bb8cf1a85ed7 parsl.dataflow.memoization INFO: Task 2127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2127 try 0 launched on executor multi with executor id 2128 parsl.dataflow.dflow INFO: Standard output for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/83a5f775-0cf5-4fb5-b210-5409be271de4_isrForCrosstalkSources_1031721_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/83a5f775-0cf5-4fb5-b210-5409be271de4_isrForCrosstalkSources_1031721_31.stderr parsl.dataflow.dflow DEBUG: Task 2128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2128 has memoization hash e9020c74f5f9f37ad3ee4b2a749c219b parsl.dataflow.memoization INFO: Task 2128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2128 try 0 launched on executor multi with executor id 2129 parsl.dataflow.dflow INFO: Standard output for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3ee931aa-0af0-4422-979c-b237ac915a8e_isrForCrosstalkSources_1031735_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3ee931aa-0af0-4422-979c-b237ac915a8e_isrForCrosstalkSources_1031735_11.stderr parsl.dataflow.dflow DEBUG: Task 2129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2129 has memoization hash c9e0bffb9986b3bc15fe871e7513f34a parsl.dataflow.memoization INFO: Task 2129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6054e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2129 try 0 launched on executor multi with executor id 2130 parsl.dataflow.dflow INFO: Standard output for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/fa4849dc-ade4-46ca-ba74-b01c0aa8c523_isrForCrosstalkSources_1031731_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/fa4849dc-ade4-46ca-ba74-b01c0aa8c523_isrForCrosstalkSources_1031731_47.stderr parsl.dataflow.dflow DEBUG: Task 2130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2130 has memoization hash ddc531898e4f7a0f8d5d00538270cfa7 parsl.dataflow.memoization INFO: Task 2130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2130 try 0 launched on executor multi with executor id 2131 parsl.dataflow.dflow INFO: Standard output for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/37dc73a0-03b8-403e-b521-f84da81439b7_isrForCrosstalkSources_1031716_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/37dc73a0-03b8-403e-b521-f84da81439b7_isrForCrosstalkSources_1031716_5.stderr parsl.dataflow.dflow DEBUG: Task 2131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2131 has memoization hash f1509a1663c5ac6a15ab05da981a9361 parsl.dataflow.memoization INFO: Task 2131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2131 try 0 launched on executor multi with executor id 2132 parsl.dataflow.dflow INFO: Standard output for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/115e88f5-9ac5-4712-876a-bf680acae323_isrForCrosstalkSources_1031724_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/115e88f5-9ac5-4712-876a-bf680acae323_isrForCrosstalkSources_1031724_38.stderr parsl.dataflow.dflow DEBUG: Task 2132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2132 has memoization hash 360cdf5e06eff6fb5858295cb7dfd128 parsl.dataflow.memoization INFO: Task 2132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2132 try 0 launched on executor multi with executor id 2133 parsl.dataflow.dflow INFO: Standard output for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d3a8f67a-973d-459d-b76f-d1297b00ee9a_isrForCrosstalkSources_1031730_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d3a8f67a-973d-459d-b76f-d1297b00ee9a_isrForCrosstalkSources_1031730_36.stderr parsl.dataflow.dflow DEBUG: Task 2133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2133 has memoization hash 4f3290d5592bc7e32e627993ef678470 parsl.dataflow.memoization INFO: Task 2133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6058a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2133 try 0 launched on executor multi with executor id 2134 parsl.dataflow.dflow INFO: Standard output for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/715a59eb-0a9e-406c-b843-25786c2c1844_isrForCrosstalkSources_1031701_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/715a59eb-0a9e-406c-b843-25786c2c1844_isrForCrosstalkSources_1031701_25.stderr parsl.dataflow.dflow DEBUG: Task 2134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2134 has memoization hash f71fb6fd611e3ca89bdf1104f56f4faa parsl.dataflow.memoization INFO: Task 2134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6059e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2134 try 0 launched on executor multi with executor id 2135 parsl.dataflow.dflow INFO: Standard output for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/eec07911-cf50-4c60-920c-a5a3f6717894_isrForCrosstalkSources_1031715_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/eec07911-cf50-4c60-920c-a5a3f6717894_isrForCrosstalkSources_1031715_59.stderr parsl.dataflow.dflow DEBUG: Task 2135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2135 has memoization hash 4c91f719df6a5cf638c2b13a622b7a6b parsl.dataflow.memoization INFO: Task 2135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2135 try 0 launched on executor multi with executor id 2136 parsl.dataflow.dflow INFO: Standard output for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/b1d8d952-0121-4c60-8656-55062eae18a4_isrForCrosstalkSources_1031738_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/b1d8d952-0121-4c60-8656-55062eae18a4_isrForCrosstalkSources_1031738_1.stderr parsl.dataflow.dflow DEBUG: Task 2136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2136 has memoization hash 2544e8b72730b7a4406aa4725bd8c006 parsl.dataflow.memoization INFO: Task 2136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2136 try 0 launched on executor multi with executor id 2137 parsl.dataflow.dflow INFO: Standard output for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b9339086-0719-4080-b02e-97354c6697b2_isrForCrosstalkSources_1031712_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b9339086-0719-4080-b02e-97354c6697b2_isrForCrosstalkSources_1031712_60.stderr parsl.dataflow.dflow DEBUG: Task 2137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2137 has memoization hash 34b677e7e4d422b221e374a8e15850db parsl.dataflow.memoization INFO: Task 2137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2137 try 0 launched on executor multi with executor id 2138 parsl.dataflow.dflow INFO: Standard output for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/e6bd5e9d-fec3-4019-a560-a267dc883ae4_isrForCrosstalkSources_1031699_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/e6bd5e9d-fec3-4019-a560-a267dc883ae4_isrForCrosstalkSources_1031699_12.stderr parsl.dataflow.dflow DEBUG: Task 2138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2138 has memoization hash fe3d0cc922f353e05c527a3f21f175bd parsl.dataflow.memoization INFO: Task 2138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2138 try 0 launched on executor multi with executor id 2139 parsl.dataflow.dflow INFO: Standard output for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/5969260f-07c1-4e3e-b5ca-4c3779bb09e5_isrForCrosstalkSources_1031729_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/5969260f-07c1-4e3e-b5ca-4c3779bb09e5_isrForCrosstalkSources_1031729_21.stderr parsl.dataflow.dflow DEBUG: Task 2139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2139 has memoization hash a27bfd02a6366aed533250acbb46215b parsl.dataflow.memoization INFO: Task 2139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2139 try 0 launched on executor multi with executor id 2140 parsl.dataflow.dflow INFO: Standard output for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/436f4b0c-ecbd-4f46-8c2e-cea3bfeb81ae_isrForCrosstalkSources_1031735_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/436f4b0c-ecbd-4f46-8c2e-cea3bfeb81ae_isrForCrosstalkSources_1031735_28.stderr parsl.dataflow.dflow DEBUG: Task 2140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2140 has memoization hash 252d89d1bef58906eebc2369c3cc58eb parsl.dataflow.memoization INFO: Task 2140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2140 try 0 launched on executor multi with executor id 2141 parsl.dataflow.dflow INFO: Standard output for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/3716cb6d-ea99-46e7-95a1-3769c32d44ac_isrForCrosstalkSources_1031699_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/3716cb6d-ea99-46e7-95a1-3769c32d44ac_isrForCrosstalkSources_1031699_56.stderr parsl.dataflow.dflow DEBUG: Task 2141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2141 has memoization hash d1a3ddfdfe053506fc854367acedabec parsl.dataflow.memoization INFO: Task 2141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2141 try 0 launched on executor multi with executor id 2142 parsl.dataflow.dflow INFO: Standard output for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/c424b95e-3404-4b01-b6a3-ade2c10483b9_isrForCrosstalkSources_1031704_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/c424b95e-3404-4b01-b6a3-ade2c10483b9_isrForCrosstalkSources_1031704_37.stderr parsl.dataflow.dflow DEBUG: Task 2142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2142 has memoization hash fba29e75896a37ec86fd0a79b630a174 parsl.dataflow.memoization INFO: Task 2142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2142 try 0 launched on executor multi with executor id 2143 parsl.dataflow.dflow INFO: Standard output for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/47a22ead-5c98-4a19-a28e-c0a058c34b18_isrForCrosstalkSources_1031715_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/47a22ead-5c98-4a19-a28e-c0a058c34b18_isrForCrosstalkSources_1031715_11.stderr parsl.dataflow.dflow DEBUG: Task 2143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2143 has memoization hash e84e1d1cb1c78866b6da54b7fa563a62 parsl.dataflow.memoization INFO: Task 2143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f605ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3435e940d05aba31cc3c28a80f1f1016 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/38fc7250-ec9e-4ce4-8fa2-39f2110e18f2_isrForCrosstalkSources_1031720_20.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/38fc7250-ec9e-4ce4-8fa2-39f2110e18f2_isrForCrosstalkSources_1031720_20.stderr parsl.dataflow.dflow INFO: Parsl task 2143 try 0 launched on executor multi with executor id 2144 parsl.dataflow.dflow INFO: Standard output for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/112b4d84-bebb-4459-ac22-6a2eca1ed929_isrForCrosstalkSources_1031715_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/112b4d84-bebb-4459-ac22-6a2eca1ed929_isrForCrosstalkSources_1031715_29.stderr parsl.dataflow.dflow DEBUG: Task 2144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2144 has memoization hash df0d2c43077c785d90ab174c1dd1167c parsl.dataflow.memoization INFO: Task 2144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2144 try 0 launched on executor multi with executor id 2145 parsl.dataflow.dflow INFO: Standard output for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8dd7d03e-1862-4297-93a6-fccc6c69871a_isrForCrosstalkSources_1031736_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8dd7d03e-1862-4297-93a6-fccc6c69871a_isrForCrosstalkSources_1031736_3.stderr parsl.dataflow.dflow DEBUG: Task 2145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2145 has memoization hash f70b18fcba228b06b8f7e8e880a05c99 parsl.dataflow.memoization INFO: Task 2145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f604040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2145 try 0 launched on executor multi with executor id 2146 parsl.dataflow.dflow INFO: Standard output for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ac1fb40e-00c8-4d6a-95fb-0cdcfd47b07b_isrForCrosstalkSources_1031711_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ac1fb40e-00c8-4d6a-95fb-0cdcfd47b07b_isrForCrosstalkSources_1031711_46.stderr parsl.dataflow.dflow DEBUG: Task 2146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2146 has memoization hash 9f0ee969a1b9c1ade8147716b0e2fafd parsl.dataflow.memoization INFO: Task 2146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2146 try 0 launched on executor multi with executor id 2147 parsl.dataflow.dflow INFO: Standard output for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/49c9e811-ce4c-49c7-af20-98b27f304db3_isrForCrosstalkSources_1031730_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/49c9e811-ce4c-49c7-af20-98b27f304db3_isrForCrosstalkSources_1031730_21.stderr parsl.dataflow.dflow DEBUG: Task 2147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2147 has memoization hash 8cca593304c1ff128cef9948f05e6687 parsl.dataflow.memoization INFO: Task 2147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2147 try 0 launched on executor multi with executor id 2148 parsl.dataflow.dflow INFO: Standard output for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/16959245-b091-488f-a363-4f33e05bb784_isrForCrosstalkSources_1031705_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/16959245-b091-488f-a363-4f33e05bb784_isrForCrosstalkSources_1031705_59.stderr parsl.dataflow.dflow DEBUG: Task 2148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2148 has memoization hash d28a1c7438712a96bc763ae60083efe9 parsl.dataflow.memoization INFO: Task 2148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6060c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2148 try 0 launched on executor multi with executor id 2149 parsl.dataflow.dflow INFO: Standard output for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/071d7eda-70c9-4a89-868a-dd16bbcdc7ba_isrForCrosstalkSources_1031729_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/071d7eda-70c9-4a89-868a-dd16bbcdc7ba_isrForCrosstalkSources_1031729_4.stderr parsl.dataflow.dflow DEBUG: Task 2149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2149 has memoization hash 69319ab0e011a9a007b07080732dd4ab parsl.dataflow.memoization INFO: Task 2149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2149 try 0 launched on executor multi with executor id 2150 parsl.dataflow.dflow INFO: Standard output for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/382edb67-a3a3-43ac-8b41-0b33f5fb9ea0_isrForCrosstalkSources_1031705_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/382edb67-a3a3-43ac-8b41-0b33f5fb9ea0_isrForCrosstalkSources_1031705_12.stderr parsl.dataflow.dflow DEBUG: Task 2150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2150 has memoization hash 10bc0e93787855a016aba612dc531d6f parsl.dataflow.memoization INFO: Task 2150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2150 try 0 launched on executor multi with executor id 2151 parsl.dataflow.dflow INFO: Standard output for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fa265ecd-0e58-4582-9089-9050ee6c2fc0_isrForCrosstalkSources_1031702_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fa265ecd-0e58-4582-9089-9050ee6c2fc0_isrForCrosstalkSources_1031702_19.stderr parsl.dataflow.dflow DEBUG: Task 2151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2151 has memoization hash 33a300b9fd84f6a1794cbafc54c1b19e parsl.dataflow.memoization INFO: Task 2151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2151 try 0 launched on executor multi with executor id 2152 parsl.dataflow.dflow INFO: Standard output for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/432e2d7b-cb8d-4842-b9e9-55e0a96839ef_isrForCrosstalkSources_1031705_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/432e2d7b-cb8d-4842-b9e9-55e0a96839ef_isrForCrosstalkSources_1031705_5.stderr parsl.dataflow.dflow DEBUG: Task 2152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2152 has memoization hash 1f51be246abcf4f8be09e7a498fab0c6 parsl.dataflow.memoization INFO: Task 2152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6065c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2152 try 0 launched on executor multi with executor id 2153 parsl.dataflow.dflow INFO: Standard output for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/15a99062-99bc-4982-b39b-105df4d0effc_isrForCrosstalkSources_1031731_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/15a99062-99bc-4982-b39b-105df4d0effc_isrForCrosstalkSources_1031731_52.stderr parsl.dataflow.dflow DEBUG: Task 2153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2153 has memoization hash a97c0f78b8db681dd4a0632e280d9ecc parsl.dataflow.memoization INFO: Task 2153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a97faf93f584d0c7478f14733b3fffa5 with result from task 6 parsl.dataflow.dflow INFO: Parsl task 2153 try 0 launched on executor multi with executor id 2154 parsl.dataflow.dflow INFO: Standard output for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/f14b5276-93d7-4d8b-bb95-3bb1a0fc2f88_isrForCrosstalkSources_1031708_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/f14b5276-93d7-4d8b-bb95-3bb1a0fc2f88_isrForCrosstalkSources_1031708_11.stderr parsl.dataflow.dflow DEBUG: Task 2154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2154 has memoization hash 1c79e8547e3bb3d18df90f86dfa00238 parsl.dataflow.memoization INFO: Task 2154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/42d90be2-854e-4beb-a528-40acfcd6c0b3_isrForCrosstalkSources_1031708_35.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/42d90be2-854e-4beb-a528-40acfcd6c0b3_isrForCrosstalkSources_1031708_35.stderr parsl.dataflow.dflow INFO: Parsl task 2154 try 0 launched on executor multi with executor id 2155 parsl.dataflow.dflow INFO: Standard output for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c163dd45-3a39-4750-a2b4-8e4e7f80a16d_isrForCrosstalkSources_1031732_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c163dd45-3a39-4750-a2b4-8e4e7f80a16d_isrForCrosstalkSources_1031732_9.stderr parsl.dataflow.dflow DEBUG: Task 2155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2155 has memoization hash 06ba796208bcedc7762275c57799c5fe parsl.dataflow.memoization INFO: Task 2155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ab070c334d3ee97dc4de9ef37bcc7ca with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e109aa99-6b9b-4a5f-84a9-0ddb55766cee_isrForCrosstalkSources_1031727_59.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e109aa99-6b9b-4a5f-84a9-0ddb55766cee_isrForCrosstalkSources_1031727_59.stderr parsl.dataflow.dflow INFO: Parsl task 2155 try 0 launched on executor multi with executor id 2156 parsl.dataflow.dflow INFO: Standard output for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/379fb331-a237-4045-a256-f0a88e35d9ef_isrForCrosstalkSources_1031733_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/379fb331-a237-4045-a256-f0a88e35d9ef_isrForCrosstalkSources_1031733_18.stderr parsl.dataflow.dflow DEBUG: Task 2156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2156 has memoization hash abf6cbe1a6a293dd043af10577d40c1d parsl.dataflow.memoization INFO: Task 2156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2156 try 0 launched on executor multi with executor id 2157 parsl.dataflow.dflow INFO: Standard output for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1a5e23cd-f8cb-4d35-bec8-4d4b933d7277_isrForCrosstalkSources_1031733_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1a5e23cd-f8cb-4d35-bec8-4d4b933d7277_isrForCrosstalkSources_1031733_45.stderr parsl.dataflow.dflow DEBUG: Task 2157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2157 has memoization hash 3724dcf152a1535de97b8dd549731d58 parsl.dataflow.memoization INFO: Task 2157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6067a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2157 try 0 launched on executor multi with executor id 2158 parsl.dataflow.dflow INFO: Standard output for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/bf99a6c2-3d16-4478-9984-8b0bd066d837_isrForCrosstalkSources_1031721_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/bf99a6c2-3d16-4478-9984-8b0bd066d837_isrForCrosstalkSources_1031721_38.stderr parsl.dataflow.dflow DEBUG: Task 2158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2158 has memoization hash f258c7d9910954b72c49fbc1d891ca13 parsl.dataflow.memoization INFO: Task 2158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2158 try 0 launched on executor multi with executor id 2159 parsl.dataflow.dflow INFO: Standard output for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2a08da7c-7fe3-422b-b32e-a6c599c58ad0_isrForCrosstalkSources_1031734_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2a08da7c-7fe3-422b-b32e-a6c599c58ad0_isrForCrosstalkSources_1031734_48.stderr parsl.dataflow.dflow DEBUG: Task 2159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2159 has memoization hash b5f4949cc62d3e9a9685538194abf344 parsl.dataflow.memoization INFO: Task 2159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2159 try 0 launched on executor multi with executor id 2160 parsl.dataflow.dflow INFO: Standard output for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/8f2a14af-6516-47d8-8274-05e999b43d52_isrForCrosstalkSources_1031725_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/8f2a14af-6516-47d8-8274-05e999b43d52_isrForCrosstalkSources_1031725_60.stderr parsl.dataflow.dflow DEBUG: Task 2160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2160 has memoization hash 92de025509e3b5223850d359b00329ab parsl.dataflow.memoization INFO: Task 2160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2160 try 0 launched on executor multi with executor id 2161 parsl.dataflow.dflow INFO: Standard output for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a8cc5183-715d-4e61-a090-3a6726162903_isrForCrosstalkSources_1031718_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a8cc5183-715d-4e61-a090-3a6726162903_isrForCrosstalkSources_1031718_12.stderr parsl.dataflow.dflow DEBUG: Task 2161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2161 has memoization hash 25b3551e7bec062386bf908e247ff897 parsl.dataflow.memoization INFO: Task 2161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2161 try 0 launched on executor multi with executor id 2162 parsl.dataflow.dflow INFO: Standard output for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/257d254d-acbd-4a6b-bfed-4a19fb9488d3_isrForCrosstalkSources_1031701_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/257d254d-acbd-4a6b-bfed-4a19fb9488d3_isrForCrosstalkSources_1031701_26.stderr parsl.dataflow.dflow DEBUG: Task 2162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2162 has memoization hash 94d25d23df7bb82dbb9fdbc7b7346d14 parsl.dataflow.memoization INFO: Task 2162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2162 try 0 launched on executor multi with executor id 2163 parsl.dataflow.dflow INFO: Standard output for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/3d56dc7d-44a5-4aac-9c21-a1bc4650e6ca_isrForCrosstalkSources_1031737_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/3d56dc7d-44a5-4aac-9c21-a1bc4650e6ca_isrForCrosstalkSources_1031737_40.stderr parsl.dataflow.dflow DEBUG: Task 2163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2163 has memoization hash a17071b491f7087c27f86cf37fb969c0 parsl.dataflow.memoization INFO: Task 2163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2163 try 0 launched on executor multi with executor id 2164 parsl.dataflow.dflow INFO: Standard output for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/40a6c531-e4e6-4fee-be7e-c0b004941cb9_isrForCrosstalkSources_1031729_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/40a6c531-e4e6-4fee-be7e-c0b004941cb9_isrForCrosstalkSources_1031729_50.stderr parsl.dataflow.dflow DEBUG: Task 2164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2164 has memoization hash a802079ba4b1778c0c3f0d58be9d1720 parsl.dataflow.memoization INFO: Task 2164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2164 try 0 launched on executor multi with executor id 2165 parsl.dataflow.dflow INFO: Standard output for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8c91f415-5d88-4c6a-98da-3457a63518c9_isrForCrosstalkSources_1031707_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8c91f415-5d88-4c6a-98da-3457a63518c9_isrForCrosstalkSources_1031707_22.stderr parsl.dataflow.dflow DEBUG: Task 2165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2165 has memoization hash 5a5196b8495ee3e5a76f8622ce625b7d parsl.dataflow.memoization INFO: Task 2165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2165 try 0 launched on executor multi with executor id 2166 parsl.dataflow.dflow INFO: Standard output for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/6cdab13f-4fe8-4840-a84d-cdb7b0de9a21_isrForCrosstalkSources_1031702_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/6cdab13f-4fe8-4840-a84d-cdb7b0de9a21_isrForCrosstalkSources_1031702_10.stderr parsl.dataflow.dflow DEBUG: Task 2166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2166 has memoization hash f76b4af25d69ac1ea63de156b83a1215 parsl.dataflow.memoization INFO: Task 2166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2166 try 0 launched on executor multi with executor id 2167 parsl.dataflow.dflow INFO: Standard output for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d9e38d54-f535-4223-83b4-2c0aaf4c0af3_isrForCrosstalkSources_1031738_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d9e38d54-f535-4223-83b4-2c0aaf4c0af3_isrForCrosstalkSources_1031738_20.stderr parsl.dataflow.dflow DEBUG: Task 2167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2167 has memoization hash 00bfe3f3910bcaec7ff09b015f789364 parsl.dataflow.memoization INFO: Task 2167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2167 try 0 launched on executor multi with executor id 2168 parsl.dataflow.dflow INFO: Standard output for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ae7323b4-5495-4619-be04-575fd11a2b4e_isrForCrosstalkSources_1031699_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ae7323b4-5495-4619-be04-575fd11a2b4e_isrForCrosstalkSources_1031699_19.stderr parsl.dataflow.dflow DEBUG: Task 2168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2168 has memoization hash d74f8eeee4f216a1bc895bf8b3ae3bcb parsl.dataflow.memoization INFO: Task 2168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2168 try 0 launched on executor multi with executor id 2169 parsl.dataflow.dflow INFO: Standard output for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/55d2a2c8-5092-4243-8f7d-24ba30ec6a16_isrForCrosstalkSources_1031731_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/55d2a2c8-5092-4243-8f7d-24ba30ec6a16_isrForCrosstalkSources_1031731_2.stderr parsl.dataflow.dflow DEBUG: Task 2169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2169 has memoization hash e84934c435346947281c3e7a58da3d7a parsl.dataflow.memoization INFO: Task 2169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2169 try 0 launched on executor multi with executor id 2170 parsl.dataflow.dflow INFO: Standard output for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/326807c7-bd57-4052-8612-02abce1859a7_isrForCrosstalkSources_1031701_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/326807c7-bd57-4052-8612-02abce1859a7_isrForCrosstalkSources_1031701_43.stderr parsl.dataflow.dflow DEBUG: Task 2170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2170 has memoization hash 3c22b01ed796576c695399ebf895e636 parsl.dataflow.memoization INFO: Task 2170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2170 try 0 launched on executor multi with executor id 2171 parsl.dataflow.dflow INFO: Standard output for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f06a35e1-89f8-433b-b0e6-b055fb79e234_isrForCrosstalkSources_1031699_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f06a35e1-89f8-433b-b0e6-b055fb79e234_isrForCrosstalkSources_1031699_40.stderr parsl.dataflow.dflow DEBUG: Task 2171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2171 has memoization hash 2d1d5b615806ed4946f5b7573b5663fc parsl.dataflow.memoization INFO: Task 2171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2171 try 0 launched on executor multi with executor id 2172 parsl.dataflow.dflow INFO: Standard output for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/cfc1dfca-0103-4f02-a855-589b06b72404_isrForCrosstalkSources_1031706_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/cfc1dfca-0103-4f02-a855-589b06b72404_isrForCrosstalkSources_1031706_34.stderr parsl.dataflow.dflow DEBUG: Task 2172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2172 has memoization hash 618356e019f570a639d0b8280f78f1d8 parsl.dataflow.memoization INFO: Task 2172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2172 try 0 launched on executor multi with executor id 2173 parsl.dataflow.dflow INFO: Standard output for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/cb666c2e-874c-493b-977b-239691945246_isrForCrosstalkSources_1031720_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/cb666c2e-874c-493b-977b-239691945246_isrForCrosstalkSources_1031720_25.stderr parsl.dataflow.dflow DEBUG: Task 2173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2173 has memoization hash 2ac4d6bf6da4ab5521810def17e266e0 parsl.dataflow.memoization INFO: Task 2173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2173 try 0 launched on executor multi with executor id 2174 parsl.dataflow.dflow INFO: Standard output for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/672fb773-80f4-4a8a-b22d-30a954a4c7e9_isrForCrosstalkSources_1031701_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/672fb773-80f4-4a8a-b22d-30a954a4c7e9_isrForCrosstalkSources_1031701_54.stderr parsl.dataflow.dflow DEBUG: Task 2174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2174 has memoization hash 399852d6e48f213686cf31dbf2430285 parsl.dataflow.memoization INFO: Task 2174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6072e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2174 try 0 launched on executor multi with executor id 2175 parsl.dataflow.dflow INFO: Standard output for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/785fb7ef-7978-495c-b5c9-d80f561d7158_isrForCrosstalkSources_1031734_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/785fb7ef-7978-495c-b5c9-d80f561d7158_isrForCrosstalkSources_1031734_38.stderr parsl.dataflow.dflow DEBUG: Task 2175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2175 has memoization hash a32a3fa9ec79e03531831b1cc8f7b8db parsl.dataflow.memoization INFO: Task 2175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2175 try 0 launched on executor multi with executor id 2176 parsl.dataflow.dflow INFO: Standard output for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/fdb9408b-2415-4a78-823d-4cfe3f840859_isrForCrosstalkSources_1031724_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/fdb9408b-2415-4a78-823d-4cfe3f840859_isrForCrosstalkSources_1031724_25.stderr parsl.dataflow.dflow DEBUG: Task 2176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2176 has memoization hash b597591439fea4d7b2c741e392dace15 parsl.dataflow.memoization INFO: Task 2176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2176 try 0 launched on executor multi with executor id 2177 parsl.dataflow.dflow INFO: Standard output for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8ac33128-0e7d-47ba-848e-c735d606de69_isrForCrosstalkSources_1031709_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8ac33128-0e7d-47ba-848e-c735d606de69_isrForCrosstalkSources_1031709_57.stderr parsl.dataflow.dflow DEBUG: Task 2177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2177 has memoization hash 313016bffcb9c48bb58e7d0bd7bdb87b parsl.dataflow.memoization INFO: Task 2177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:23 proc_lsst.multi:170 [INFO] [multi] job_id 17147957 proc_lsst.multi INFO: [multi] job_id 17147957 2024-03-17 04:02:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-17 04:02:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147957 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147957 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17147957 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-17 04:02:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-17 04:02:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-17 04:02:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2177 try 0 launched on executor multi with executor id 2178 parsl.dataflow.dflow INFO: Standard output for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c3e735e1-5c16-4f64-8efc-cea107f0c10b_isrForCrosstalkSources_1031716_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c3e735e1-5c16-4f64-8efc-cea107f0c10b_isrForCrosstalkSources_1031716_18.stderr parsl.dataflow.dflow DEBUG: Task 2178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2178 has memoization hash ddfaa8950028111a78c44b4376fa0af3 parsl.dataflow.memoization INFO: Task 2178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f773f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2178 try 0 launched on executor multi with executor id 2179 parsl.dataflow.dflow INFO: Standard output for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/611790c3-e9ae-426f-a212-9302e9fee8d0_isrForCrosstalkSources_1031705_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/611790c3-e9ae-426f-a212-9302e9fee8d0_isrForCrosstalkSources_1031705_40.stderr parsl.dataflow.dflow DEBUG: Task 2179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2179 has memoization hash 40bbb7e54b43cc5cbb5fcf141ac59109 parsl.dataflow.memoization INFO: Task 2179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2179 try 0 launched on executor multi with executor id 2180 parsl.dataflow.dflow INFO: Standard output for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/2d76a4c6-d5d5-4f52-9d70-958b8ec1c1c7_isrForCrosstalkSources_1031720_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/2d76a4c6-d5d5-4f52-9d70-958b8ec1c1c7_isrForCrosstalkSources_1031720_51.stderr parsl.dataflow.dflow DEBUG: Task 2180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2180 has memoization hash 2a3b93cd4d64b4590de005e208d76e4c parsl.dataflow.memoization INFO: Task 2180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f772ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2180 try 0 launched on executor multi with executor id 2181 parsl.dataflow.dflow INFO: Standard output for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a321d178-b0e1-4583-837f-c89a250a9026_isrForCrosstalkSources_1031706_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a321d178-b0e1-4583-837f-c89a250a9026_isrForCrosstalkSources_1031706_5.stderr parsl.dataflow.dflow DEBUG: Task 2181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2181 has memoization hash b4308ec69a70e7343c0e9b53d4db9d4a parsl.dataflow.memoization INFO: Task 2181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6074c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2181 try 0 launched on executor multi with executor id 2182 parsl.dataflow.dflow INFO: Standard output for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/00968709-f2f2-4e9f-b7f3-e5d55db8a772_isrForCrosstalkSources_1031714_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/00968709-f2f2-4e9f-b7f3-e5d55db8a772_isrForCrosstalkSources_1031714_13.stderr parsl.dataflow.dflow DEBUG: Task 2182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2182 has memoization hash 68940d46fddd4caa950d02f700980d54 parsl.dataflow.memoization INFO: Task 2182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2182 try 0 launched on executor multi with executor id 2183 parsl.dataflow.dflow INFO: Standard output for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/749eebc9-e5e3-4a0a-9468-4eb6a1204c02_isrForCrosstalkSources_1031722_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/749eebc9-e5e3-4a0a-9468-4eb6a1204c02_isrForCrosstalkSources_1031722_19.stderr parsl.dataflow.dflow DEBUG: Task 2183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2183 has memoization hash 96504846fc4f07c10e2ab0c331832830 parsl.dataflow.memoization INFO: Task 2183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2183 try 0 launched on executor multi with executor id 2184 parsl.dataflow.dflow INFO: Standard output for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/862296ea-1762-4fb6-88ae-dfa23ef79e9f_isrForCrosstalkSources_1031716_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/862296ea-1762-4fb6-88ae-dfa23ef79e9f_isrForCrosstalkSources_1031716_54.stderr parsl.dataflow.dflow DEBUG: Task 2184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2184 has memoization hash d4dbfe02a18f343a2f95e7b238c0852f parsl.dataflow.memoization INFO: Task 2184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2184 try 0 launched on executor multi with executor id 2185 parsl.dataflow.dflow INFO: Standard output for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0ba34f58-9c07-4d8f-a8d7-48d0db714ff5_isrForCrosstalkSources_1031732_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0ba34f58-9c07-4d8f-a8d7-48d0db714ff5_isrForCrosstalkSources_1031732_62.stderr parsl.dataflow.dflow DEBUG: Task 2185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2185 has memoization hash d949c5ac197fda9ce9c65d726ef53f60 parsl.dataflow.memoization INFO: Task 2185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2185 try 0 launched on executor multi with executor id 2186 parsl.dataflow.dflow INFO: Standard output for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/8cd39082-900d-4848-ac26-caa99530fa0d_isrForCrosstalkSources_1031737_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/8cd39082-900d-4848-ac26-caa99530fa0d_isrForCrosstalkSources_1031737_47.stderr parsl.dataflow.dflow DEBUG: Task 2186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2186 has memoization hash e87b09e63edf063b28dce971758b39f1 parsl.dataflow.memoization INFO: Task 2186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2186 try 0 launched on executor multi with executor id 2187 parsl.dataflow.dflow INFO: Standard output for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/02b4a96d-e856-4012-ba28-9c1996a81be0_isrForCrosstalkSources_1031708_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/02b4a96d-e856-4012-ba28-9c1996a81be0_isrForCrosstalkSources_1031708_31.stderr parsl.dataflow.dflow DEBUG: Task 2187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2187 has memoization hash 3e56116ca768ab2981cb75ba47f0196f parsl.dataflow.memoization INFO: Task 2187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2187 try 0 launched on executor multi with executor id 2188 parsl.dataflow.dflow INFO: Standard output for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/6e03efbc-1341-468f-a262-dc57d11407e0_isrForCrosstalkSources_1031706_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/6e03efbc-1341-468f-a262-dc57d11407e0_isrForCrosstalkSources_1031706_56.stderr parsl.dataflow.dflow DEBUG: Task 2188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2188 has memoization hash 44fccafda5f71bc7c84393d4c6ad7f05 parsl.dataflow.memoization INFO: Task 2188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2188 try 0 launched on executor multi with executor id 2189 parsl.dataflow.dflow INFO: Standard output for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a88eedd3-418d-4b3d-bfdc-8b502d412261_isrForCrosstalkSources_1031702_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a88eedd3-418d-4b3d-bfdc-8b502d412261_isrForCrosstalkSources_1031702_45.stderr parsl.dataflow.dflow DEBUG: Task 2189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2189 has memoization hash 1576da178e51ac6ac8665f42dd8270db parsl.dataflow.memoization INFO: Task 2189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2189 try 0 launched on executor multi with executor id 2190 parsl.dataflow.dflow INFO: Standard output for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/41590ab8-6ad9-4257-9ff5-701af7c309ad_isrForCrosstalkSources_1031725_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/41590ab8-6ad9-4257-9ff5-701af7c309ad_isrForCrosstalkSources_1031725_39.stderr parsl.dataflow.dflow DEBUG: Task 2190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2190 has memoization hash ae247c8debed6efdee51ffc2adc6260b parsl.dataflow.memoization INFO: Task 2190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2190 try 0 launched on executor multi with executor id 2191 parsl.dataflow.dflow INFO: Standard output for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d197296f-b38c-4c86-b445-9fe1001bb815_isrForCrosstalkSources_1031701_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d197296f-b38c-4c86-b445-9fe1001bb815_isrForCrosstalkSources_1031701_14.stderr parsl.dataflow.dflow DEBUG: Task 2191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2191 has memoization hash 3a520d38cf694e1642c6e69f9cbc17d7 parsl.dataflow.memoization INFO: Task 2191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2191 try 0 launched on executor multi with executor id 2192 parsl.dataflow.dflow INFO: Standard output for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/09cd52a6-c8fd-4fa5-bb7b-0b77cc0a9f4f_isrForCrosstalkSources_1031721_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/09cd52a6-c8fd-4fa5-bb7b-0b77cc0a9f4f_isrForCrosstalkSources_1031721_37.stderr parsl.dataflow.dflow DEBUG: Task 2192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2192 has memoization hash f4485e5147c60b8aed41f803786bb0b6 parsl.dataflow.memoization INFO: Task 2192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2192 try 0 launched on executor multi with executor id 2193 parsl.dataflow.dflow INFO: Standard output for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/56bdbe1a-85b9-4e3f-b32b-3bf8c2aeaa70_isrForCrosstalkSources_1031723_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/56bdbe1a-85b9-4e3f-b32b-3bf8c2aeaa70_isrForCrosstalkSources_1031723_17.stderr parsl.dataflow.dflow DEBUG: Task 2193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2193 has memoization hash 9bd058952064f0273aba1d336e8897ee parsl.dataflow.memoization INFO: Task 2193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2193 try 0 launched on executor multi with executor id 2194 parsl.dataflow.dflow INFO: Standard output for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a2109690-debd-4783-a7cb-99bdd07d803f_isrForCrosstalkSources_1031723_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a2109690-debd-4783-a7cb-99bdd07d803f_isrForCrosstalkSources_1031723_24.stderr parsl.dataflow.dflow DEBUG: Task 2194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2194 has memoization hash 6f093bf5afce5a80a6067ed211f44dd7 parsl.dataflow.memoization INFO: Task 2194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2194 try 0 launched on executor multi with executor id 2195 parsl.dataflow.dflow INFO: Standard output for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/5d85f309-5c4c-4efc-a3a2-be1caba76488_isrForCrosstalkSources_1031733_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/5d85f309-5c4c-4efc-a3a2-be1caba76488_isrForCrosstalkSources_1031733_42.stderr parsl.dataflow.dflow DEBUG: Task 2195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2195 has memoization hash db45d6e3e75a48d4a90606d058320e0e parsl.dataflow.memoization INFO: Task 2195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2195 try 0 launched on executor multi with executor id 2196 parsl.dataflow.dflow INFO: Standard output for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c2399dfa-14c9-4666-ba5c-dbf8f6876920_isrForCrosstalkSources_1031719_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c2399dfa-14c9-4666-ba5c-dbf8f6876920_isrForCrosstalkSources_1031719_28.stderr parsl.dataflow.dflow DEBUG: Task 2196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2196 has memoization hash 00c15b21d0af7d4e6a0a931e1128992c parsl.dataflow.memoization INFO: Task 2196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6900e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2196 try 0 launched on executor multi with executor id 2197 parsl.dataflow.dflow INFO: Standard output for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/35aededa-88b0-4be7-b1dd-f83f2b0b5d3f_isrForCrosstalkSources_1031701_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/35aededa-88b0-4be7-b1dd-f83f2b0b5d3f_isrForCrosstalkSources_1031701_59.stderr parsl.dataflow.dflow DEBUG: Task 2197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2197 has memoization hash ebadad6979ded823dc1efa82a78ee0ce parsl.dataflow.memoization INFO: Task 2197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2197 try 0 launched on executor multi with executor id 2198 parsl.dataflow.dflow INFO: Standard output for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/4772ee43-a628-4ef0-91ef-859e477be4f4_isrForCrosstalkSources_1031731_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/4772ee43-a628-4ef0-91ef-859e477be4f4_isrForCrosstalkSources_1031731_59.stderr parsl.dataflow.dflow DEBUG: Task 2198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2198 has memoization hash f126a27cfabdf6228c484c789255b509 parsl.dataflow.memoization INFO: Task 2198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2198 try 0 launched on executor multi with executor id 2199 parsl.dataflow.dflow INFO: Standard output for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/e1c2f7eb-2291-4449-b2c7-c857cb505a06_isrForCrosstalkSources_1031734_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/e1c2f7eb-2291-4449-b2c7-c857cb505a06_isrForCrosstalkSources_1031734_36.stderr parsl.dataflow.dflow DEBUG: Task 2199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2199 has memoization hash 6ca26a75c410b6b7196ea8c6e7c0c25a parsl.dataflow.memoization INFO: Task 2199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6902c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2199 try 0 launched on executor multi with executor id 2200 parsl.dataflow.dflow INFO: Standard output for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/514dcb61-c15a-4961-af5c-0f7669f8716c_isrForCrosstalkSources_1031718_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/514dcb61-c15a-4961-af5c-0f7669f8716c_isrForCrosstalkSources_1031718_44.stderr parsl.dataflow.dflow DEBUG: Task 2200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2200 has memoization hash de797ccd09787b79a8bc3c56fc053447 parsl.dataflow.memoization INFO: Task 2200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2200 try 0 launched on executor multi with executor id 2201 parsl.dataflow.dflow INFO: Standard output for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/932a3e00-aa98-4f22-9a70-1e2d3443a355_isrForCrosstalkSources_1031706_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/932a3e00-aa98-4f22-9a70-1e2d3443a355_isrForCrosstalkSources_1031706_55.stderr parsl.dataflow.dflow DEBUG: Task 2201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2201 has memoization hash 18c2341356535039149e7aa8dbe8c5b1 parsl.dataflow.memoization INFO: Task 2201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2201 try 0 launched on executor multi with executor id 2202 parsl.dataflow.dflow INFO: Standard output for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/e23dd587-c20d-438a-85dd-ef690b6cc004_isrForCrosstalkSources_1031729_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/e23dd587-c20d-438a-85dd-ef690b6cc004_isrForCrosstalkSources_1031729_61.stderr parsl.dataflow.dflow DEBUG: Task 2202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2202 has memoization hash faf4f75d58a6d54f448bee608f7b16a8 parsl.dataflow.memoization INFO: Task 2202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2202 try 0 launched on executor multi with executor id 2203 parsl.dataflow.dflow INFO: Standard output for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8525da9d-debb-4e9c-b789-73606c83a78a_isrForCrosstalkSources_1031719_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8525da9d-debb-4e9c-b789-73606c83a78a_isrForCrosstalkSources_1031719_13.stderr parsl.dataflow.dflow DEBUG: Task 2203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2203 has memoization hash 27f554581fb15a5b3f129cbd78ef0b7d parsl.dataflow.memoization INFO: Task 2203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6904a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2203 try 0 launched on executor multi with executor id 2204 parsl.dataflow.dflow INFO: Standard output for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/157743bb-34c5-4e12-8b41-666f0a846bfa_isrForCrosstalkSources_1031721_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/157743bb-34c5-4e12-8b41-666f0a846bfa_isrForCrosstalkSources_1031721_50.stderr parsl.dataflow.dflow DEBUG: Task 2204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2204 has memoization hash 29f3da789f2eb85c7ccb26ccacc0a955 parsl.dataflow.memoization INFO: Task 2204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6905e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2204 try 0 launched on executor multi with executor id 2205 parsl.dataflow.dflow INFO: Standard output for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e88be92b-e68c-43b3-87d5-8b399dc7dec1_isrForCrosstalkSources_1031728_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e88be92b-e68c-43b3-87d5-8b399dc7dec1_isrForCrosstalkSources_1031728_33.stderr parsl.dataflow.dflow DEBUG: Task 2205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2205 has memoization hash 1f9467f2f10319b55522f162647672f1 parsl.dataflow.memoization INFO: Task 2205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2205 try 0 launched on executor multi with executor id 2206 parsl.dataflow.dflow INFO: Standard output for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/59d75d27-0bf1-40af-91e7-13374e59d709_isrForCrosstalkSources_1031699_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/59d75d27-0bf1-40af-91e7-13374e59d709_isrForCrosstalkSources_1031699_58.stderr parsl.dataflow.dflow DEBUG: Task 2206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2206 has memoization hash bf14a9a3c6981107fc3a4c4ed3a8fed2 parsl.dataflow.memoization INFO: Task 2206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2206 try 0 launched on executor multi with executor id 2207 parsl.dataflow.dflow INFO: Standard output for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/08ff2c75-ed1f-40ee-aa31-4c0b7173d528_isrForCrosstalkSources_1031728_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/08ff2c75-ed1f-40ee-aa31-4c0b7173d528_isrForCrosstalkSources_1031728_20.stderr parsl.dataflow.dflow DEBUG: Task 2207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2207 has memoization hash cf2ff582c845c2c336bca6cdc8c2258f parsl.dataflow.memoization INFO: Task 2207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2207 try 0 launched on executor multi with executor id 2208 parsl.dataflow.dflow INFO: Standard output for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8e852631-baf2-4035-b33d-a18656b3fad1_isrForCrosstalkSources_1031721_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8e852631-baf2-4035-b33d-a18656b3fad1_isrForCrosstalkSources_1031721_5.stderr parsl.dataflow.dflow DEBUG: Task 2208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2208 has memoization hash ef1ac6cc2d7fda5d355cd9a2d3344289 parsl.dataflow.memoization INFO: Task 2208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6907c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2208 try 0 launched on executor multi with executor id 2209 parsl.dataflow.dflow INFO: Standard output for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a5e97407-67d2-4fb8-a932-ed042feb1eb5_isrForCrosstalkSources_1031701_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a5e97407-67d2-4fb8-a932-ed042feb1eb5_isrForCrosstalkSources_1031701_23.stderr parsl.dataflow.dflow DEBUG: Task 2209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2209 has memoization hash 4e73cd0d8b7ad10340ed5061829eece3 parsl.dataflow.memoization INFO: Task 2209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2209 try 0 launched on executor multi with executor id 2210 parsl.dataflow.dflow INFO: Standard output for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/dac518f5-c6ad-4481-b129-6b3424f05860_isrForCrosstalkSources_1031714_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/dac518f5-c6ad-4481-b129-6b3424f05860_isrForCrosstalkSources_1031714_31.stderr parsl.dataflow.dflow DEBUG: Task 2210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2210 has memoization hash 032454ffb6dbb0333db0b73554736d12 parsl.dataflow.memoization INFO: Task 2210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6909a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2210 try 0 launched on executor multi with executor id 2211 parsl.dataflow.dflow INFO: Standard output for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7231d166-d8a5-4851-9878-ae8bf6895a02_isrForCrosstalkSources_1031714_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7231d166-d8a5-4851-9878-ae8bf6895a02_isrForCrosstalkSources_1031714_8.stderr parsl.dataflow.dflow DEBUG: Task 2211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2211 has memoization hash a265f207ff8779b151abeb3e3b4124e2 parsl.dataflow.memoization INFO: Task 2211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2211 try 0 launched on executor multi with executor id 2212 parsl.dataflow.dflow INFO: Standard output for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d1e9afa1-cef9-4674-974b-0ebdb49968d4_isrForCrosstalkSources_1031715_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d1e9afa1-cef9-4674-974b-0ebdb49968d4_isrForCrosstalkSources_1031715_32.stderr parsl.dataflow.dflow DEBUG: Task 2212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2212 has memoization hash b5fd9999a78ffac4a99b2e379f94231f parsl.dataflow.memoization INFO: Task 2212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2212 try 0 launched on executor multi with executor id 2213 parsl.dataflow.dflow INFO: Standard output for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/33465f59-5aeb-4a81-aba2-a9ed2e1ac896_isrForCrosstalkSources_1031717_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/33465f59-5aeb-4a81-aba2-a9ed2e1ac896_isrForCrosstalkSources_1031717_32.stderr parsl.dataflow.dflow DEBUG: Task 2213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2213 has memoization hash 9f1b86c8c8b2e71debac463665e6edaf parsl.dataflow.memoization INFO: Task 2213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2213 try 0 launched on executor multi with executor id 2214 parsl.dataflow.dflow INFO: Standard output for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/218fa0eb-9481-458c-a629-eafa6c329047_isrForCrosstalkSources_1031738_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/218fa0eb-9481-458c-a629-eafa6c329047_isrForCrosstalkSources_1031738_18.stderr parsl.dataflow.dflow DEBUG: Task 2214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2214 has memoization hash 5179dd9ed3e2070f3311c46f74693fd4 parsl.dataflow.memoization INFO: Task 2214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2214 try 0 launched on executor multi with executor id 2215 parsl.dataflow.dflow INFO: Standard output for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/88497211-2e62-4619-b352-e626b561abca_isrForCrosstalkSources_1031712_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/88497211-2e62-4619-b352-e626b561abca_isrForCrosstalkSources_1031712_9.stderr parsl.dataflow.dflow DEBUG: Task 2215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2215 has memoization hash 80846051634fd6b9c64bbce6c8a112f0 parsl.dataflow.memoization INFO: Task 2215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2215 try 0 launched on executor multi with executor id 2216 parsl.dataflow.dflow INFO: Standard output for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/69d2cf43-223a-4583-8aad-4d84e6cc5086_isrForCrosstalkSources_1031724_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/69d2cf43-223a-4583-8aad-4d84e6cc5086_isrForCrosstalkSources_1031724_2.stderr parsl.dataflow.dflow DEBUG: Task 2216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2216 has memoization hash fc1a0242daa5ff96197dcaff5b1545fb parsl.dataflow.memoization INFO: Task 2216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2216 try 0 launched on executor multi with executor id 2217 parsl.dataflow.dflow INFO: Standard output for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8c426a0c-9495-4d4c-bdfa-8de5a822b236_isrForCrosstalkSources_1031703_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8c426a0c-9495-4d4c-bdfa-8de5a822b236_isrForCrosstalkSources_1031703_45.stderr parsl.dataflow.dflow DEBUG: Task 2217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2217 has memoization hash 9a6730c919f46e2ef3e94117197012cd parsl.dataflow.memoization INFO: Task 2217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2217 try 0 launched on executor multi with executor id 2218 parsl.dataflow.dflow INFO: Standard output for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fb92c31a-eee5-4df7-b1d8-9f0ad3742cc9_isrForCrosstalkSources_1031700_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fb92c31a-eee5-4df7-b1d8-9f0ad3742cc9_isrForCrosstalkSources_1031700_34.stderr parsl.dataflow.dflow DEBUG: Task 2218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2218 has memoization hash 9ee2b04e17e7dc9a1d699b057921bc5f parsl.dataflow.memoization INFO: Task 2218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2218 try 0 launched on executor multi with executor id 2219 parsl.dataflow.dflow INFO: Standard output for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/20adbfd3-012d-474c-84d2-685bd8d5425b_isrForCrosstalkSources_1031737_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/20adbfd3-012d-474c-84d2-685bd8d5425b_isrForCrosstalkSources_1031737_19.stderr parsl.dataflow.dflow DEBUG: Task 2219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2219 has memoization hash 1b4b83fba81edf0067c792ee070629e5 parsl.dataflow.memoization INFO: Task 2219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2219 try 0 launched on executor multi with executor id 2220 parsl.dataflow.dflow INFO: Standard output for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/69b896c2-cb76-4088-858f-75ba91c1f86a_isrForCrosstalkSources_1031727_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/69b896c2-cb76-4088-858f-75ba91c1f86a_isrForCrosstalkSources_1031727_28.stderr parsl.dataflow.dflow DEBUG: Task 2220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2220 has memoization hash 5f53d0c57667199cc408417de48b55de parsl.dataflow.memoization INFO: Task 2220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2220 try 0 launched on executor multi with executor id 2221 parsl.dataflow.dflow INFO: Standard output for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3b6c8243-21da-40fa-a405-468061d354e6_isrForCrosstalkSources_1031713_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3b6c8243-21da-40fa-a405-468061d354e6_isrForCrosstalkSources_1031713_53.stderr parsl.dataflow.dflow DEBUG: Task 2221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2221 has memoization hash 77aec0096ce57baad553b948470e8ef2 parsl.dataflow.memoization INFO: Task 2221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2221 try 0 launched on executor multi with executor id 2222 parsl.dataflow.dflow INFO: Standard output for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/2c3d36a9-d07e-4df4-a971-60aa996d0d1a_isrForCrosstalkSources_1031722_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/2c3d36a9-d07e-4df4-a971-60aa996d0d1a_isrForCrosstalkSources_1031722_31.stderr parsl.dataflow.dflow DEBUG: Task 2222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2222 has memoization hash f7c4ae7681d0f6c275085de06331e5b6 parsl.dataflow.memoization INFO: Task 2222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6911c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2222 try 0 launched on executor multi with executor id 2223 parsl.dataflow.dflow INFO: Standard output for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/506feb06-cfc1-4b42-8ffd-f70924c202a4_isrForCrosstalkSources_1031723_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/506feb06-cfc1-4b42-8ffd-f70924c202a4_isrForCrosstalkSources_1031723_51.stderr parsl.dataflow.dflow DEBUG: Task 2223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2223 has memoization hash 56ca578daf35eb45525b5596a49317af parsl.dataflow.memoization INFO: Task 2223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2223 try 0 launched on executor multi with executor id 2224 parsl.dataflow.dflow INFO: Standard output for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8702574b-ae98-4720-8a16-e39641a3c3ac_isrForCrosstalkSources_1031707_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8702574b-ae98-4720-8a16-e39641a3c3ac_isrForCrosstalkSources_1031707_48.stderr parsl.dataflow.dflow DEBUG: Task 2224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2224 has memoization hash 992a84b1b8a099d789b4662cee917ea4 parsl.dataflow.memoization INFO: Task 2224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2224 try 0 launched on executor multi with executor id 2225 parsl.dataflow.dflow INFO: Standard output for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/950e4543-777e-4891-8f2e-f9615eb73835_isrForCrosstalkSources_1031735_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/950e4543-777e-4891-8f2e-f9615eb73835_isrForCrosstalkSources_1031735_29.stderr parsl.dataflow.dflow DEBUG: Task 2225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2225 has memoization hash 9533bde41ece82b735c9109085bbf913 parsl.dataflow.memoization INFO: Task 2225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2225 try 0 launched on executor multi with executor id 2226 parsl.dataflow.dflow INFO: Standard output for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ee5dff00-507f-4bab-9c7c-005ce01d1f54_isrForCrosstalkSources_1031708_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ee5dff00-507f-4bab-9c7c-005ce01d1f54_isrForCrosstalkSources_1031708_10.stderr parsl.dataflow.dflow DEBUG: Task 2226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2226 has memoization hash a42c49ea6ef8cb80b3bd7404b23185e5 parsl.dataflow.memoization INFO: Task 2226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6913a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2226 try 0 launched on executor multi with executor id 2227 parsl.dataflow.dflow INFO: Standard output for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/375e5568-f222-40f5-938c-d01b8284153a_isrForCrosstalkSources_1031731_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/375e5568-f222-40f5-938c-d01b8284153a_isrForCrosstalkSources_1031731_20.stderr parsl.dataflow.dflow DEBUG: Task 2227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2227 has memoization hash de54e105269d18e2bfb460cc585068ac parsl.dataflow.memoization INFO: Task 2227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6914e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2227 try 0 launched on executor multi with executor id 2228 parsl.dataflow.dflow INFO: Standard output for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/883a0ff1-d6f6-4f94-a86e-38ba3f81a02e_isrForCrosstalkSources_1031735_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/883a0ff1-d6f6-4f94-a86e-38ba3f81a02e_isrForCrosstalkSources_1031735_52.stderr parsl.dataflow.dflow DEBUG: Task 2228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2228 has memoization hash 13aec746e47ce2555b92df9e7fab90c1 parsl.dataflow.memoization INFO: Task 2228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2228 try 0 launched on executor multi with executor id 2229 parsl.dataflow.dflow INFO: Standard output for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/9d7f78d5-1f6d-4e07-a1cc-cef704bd6c95_isrForCrosstalkSources_1031724_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/9d7f78d5-1f6d-4e07-a1cc-cef704bd6c95_isrForCrosstalkSources_1031724_34.stderr parsl.dataflow.dflow DEBUG: Task 2229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2229 has memoization hash bd749d1ef651033faa7cb8191649ca8c parsl.dataflow.memoization INFO: Task 2229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2229 try 0 launched on executor multi with executor id 2230 parsl.dataflow.dflow INFO: Standard output for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/fea3ae75-3d8e-4cfd-b452-bca1f65154cd_isrForCrosstalkSources_1031730_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/fea3ae75-3d8e-4cfd-b452-bca1f65154cd_isrForCrosstalkSources_1031730_34.stderr parsl.dataflow.dflow DEBUG: Task 2230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2230 has memoization hash d6c45a26b162bd1b4c7da8c53aa0c53c parsl.dataflow.memoization INFO: Task 2230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2230 try 0 launched on executor multi with executor id 2231 parsl.dataflow.dflow INFO: Standard output for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/b087c10b-225d-4c54-a0b1-342b0106ba31_isrForCrosstalkSources_1031700_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/b087c10b-225d-4c54-a0b1-342b0106ba31_isrForCrosstalkSources_1031700_10.stderr parsl.dataflow.dflow DEBUG: Task 2231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2231 has memoization hash e7da8edee20e852101fbc538f919eb4a parsl.dataflow.memoization INFO: Task 2231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6916c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2231 try 0 launched on executor multi with executor id 2232 parsl.dataflow.dflow INFO: Standard output for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/8811da29-ec3f-418a-84ff-7b4cb153108c_isrForCrosstalkSources_1031713_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/8811da29-ec3f-418a-84ff-7b4cb153108c_isrForCrosstalkSources_1031713_48.stderr parsl.dataflow.dflow DEBUG: Task 2232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2232 has memoization hash 792422fbd16892cb0f7751c47d8d4c3f parsl.dataflow.memoization INFO: Task 2232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2232 try 0 launched on executor multi with executor id 2233 parsl.dataflow.dflow INFO: Standard output for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d8328b06-d650-415c-bab0-910439a511e9_isrForCrosstalkSources_1031715_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d8328b06-d650-415c-bab0-910439a511e9_isrForCrosstalkSources_1031715_48.stderr parsl.dataflow.dflow DEBUG: Task 2233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2233 has memoization hash b277a76e84cd1545c70c1378bbcbe734 parsl.dataflow.memoization INFO: Task 2233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2233 try 0 launched on executor multi with executor id 2234 parsl.dataflow.dflow INFO: Standard output for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d8060273-2360-493e-8612-25ca1d2b32e0_isrForCrosstalkSources_1031709_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d8060273-2360-493e-8612-25ca1d2b32e0_isrForCrosstalkSources_1031709_25.stderr parsl.dataflow.dflow DEBUG: Task 2234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2234 has memoization hash 43a56d813f1a5c024e48f1edd991e1c4 parsl.dataflow.memoization INFO: Task 2234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6918a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2234 try 0 launched on executor multi with executor id 2235 parsl.dataflow.dflow INFO: Standard output for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4831358a-27de-4255-bfc1-cd7e35e1aa4b_isrForCrosstalkSources_1031719_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4831358a-27de-4255-bfc1-cd7e35e1aa4b_isrForCrosstalkSources_1031719_18.stderr parsl.dataflow.dflow DEBUG: Task 2235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2235 has memoization hash 2a9dc8d299d8038b97b20c8546505425 parsl.dataflow.memoization INFO: Task 2235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2235 try 0 launched on executor multi with executor id 2236 parsl.dataflow.dflow INFO: Standard output for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/6f669c9f-dc7f-4509-bbb8-8b4b221734a3_isrForCrosstalkSources_1031699_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/6f669c9f-dc7f-4509-bbb8-8b4b221734a3_isrForCrosstalkSources_1031699_37.stderr parsl.dataflow.dflow DEBUG: Task 2236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2236 has memoization hash d243fb657ae11da807d6749c683e42fd parsl.dataflow.memoization INFO: Task 2236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6919e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2236 try 0 launched on executor multi with executor id 2237 parsl.dataflow.dflow INFO: Standard output for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/26f82c2d-344a-4497-a3b0-aa0964d9d844_isrForCrosstalkSources_1031729_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/26f82c2d-344a-4497-a3b0-aa0964d9d844_isrForCrosstalkSources_1031729_35.stderr parsl.dataflow.dflow DEBUG: Task 2237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2237 has memoization hash 4edbfe4e42df0f97d9ddbd717ae5e675 parsl.dataflow.memoization INFO: Task 2237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2237 try 0 launched on executor multi with executor id 2238 parsl.dataflow.dflow INFO: Standard output for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bc1842bf-2c7f-42d1-806b-30395ec6c657_isrForCrosstalkSources_1031715_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bc1842bf-2c7f-42d1-806b-30395ec6c657_isrForCrosstalkSources_1031715_30.stderr parsl.dataflow.dflow DEBUG: Task 2238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2238 has memoization hash 471d1d4ca938afcf3873c556e29f5d97 parsl.dataflow.memoization INFO: Task 2238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2238 try 0 launched on executor multi with executor id 2239 parsl.dataflow.dflow INFO: Standard output for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2fb1cdae-6a87-40a4-884c-4303413c77e2_isrForCrosstalkSources_1031723_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2fb1cdae-6a87-40a4-884c-4303413c77e2_isrForCrosstalkSources_1031723_41.stderr parsl.dataflow.dflow DEBUG: Task 2239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2239 has memoization hash efc1a019b7419b07cac9d623673aa2d6 parsl.dataflow.memoization INFO: Task 2239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2239 try 0 launched on executor multi with executor id 2240 parsl.dataflow.dflow INFO: Standard output for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/afc94a00-4322-415c-a8e9-4449383a7208_isrForCrosstalkSources_1031726_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/afc94a00-4322-415c-a8e9-4449383a7208_isrForCrosstalkSources_1031726_42.stderr parsl.dataflow.dflow DEBUG: Task 2240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2240 has memoization hash 5b5c05b640d1dadde2a02508a20c84ac parsl.dataflow.memoization INFO: Task 2240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2240 try 0 launched on executor multi with executor id 2241 parsl.dataflow.dflow INFO: Standard output for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/0c99e524-1a69-4cc1-a29b-419774331711_isrForCrosstalkSources_1031726_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/0c99e524-1a69-4cc1-a29b-419774331711_isrForCrosstalkSources_1031726_26.stderr parsl.dataflow.dflow DEBUG: Task 2241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2241 has memoization hash e2d7b8c01c7ceaeb571eaa1a595e7ed5 parsl.dataflow.memoization INFO: Task 2241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2241 try 0 launched on executor multi with executor id 2242 parsl.dataflow.dflow INFO: Standard output for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5ff801d4-a4ed-4e73-8229-1a8a3ad0c96e_isrForCrosstalkSources_1031735_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5ff801d4-a4ed-4e73-8229-1a8a3ad0c96e_isrForCrosstalkSources_1031735_26.stderr parsl.dataflow.dflow DEBUG: Task 2242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2242 has memoization hash 4e76cc1d0dc18ff5596ae2e3920bf579 parsl.dataflow.memoization INFO: Task 2242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2242 try 0 launched on executor multi with executor id 2243 parsl.dataflow.dflow INFO: Standard output for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9f261731-65a1-4bea-ba34-c4286bfbfda8_isrForCrosstalkSources_1031716_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9f261731-65a1-4bea-ba34-c4286bfbfda8_isrForCrosstalkSources_1031716_2.stderr parsl.dataflow.dflow DEBUG: Task 2243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2243 has memoization hash 54352d681e3949e41d05c1764d0ef978 parsl.dataflow.memoization INFO: Task 2243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2243 try 0 launched on executor multi with executor id 2244 parsl.dataflow.dflow INFO: Standard output for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ff2592e6-ba3b-4194-88fc-9cbd70155043_isrForCrosstalkSources_1031717_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ff2592e6-ba3b-4194-88fc-9cbd70155043_isrForCrosstalkSources_1031717_44.stderr parsl.dataflow.dflow DEBUG: Task 2244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2244 has memoization hash 670a02b28b59265e48de4d8bf686f35f parsl.dataflow.memoization INFO: Task 2244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2244 try 0 launched on executor multi with executor id 2245 parsl.dataflow.dflow INFO: Standard output for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/51ec72ff-a78c-4261-876f-3bf3ea59aa07_isrForCrosstalkSources_1031730_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/51ec72ff-a78c-4261-876f-3bf3ea59aa07_isrForCrosstalkSources_1031730_37.stderr parsl.dataflow.dflow DEBUG: Task 2245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2245 has memoization hash 3a35ac5e5ce06397586c223a9bf1f628 parsl.dataflow.memoization INFO: Task 2245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f691f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2245 try 0 launched on executor multi with executor id 2246 parsl.dataflow.dflow INFO: Standard output for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/4929f560-e715-48eb-8df7-ee0f317d15c8_isrForCrosstalkSources_1031702_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/4929f560-e715-48eb-8df7-ee0f317d15c8_isrForCrosstalkSources_1031702_56.stderr parsl.dataflow.dflow DEBUG: Task 2246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2246 has memoization hash 4e200c77444d5e3262cacca10f200c2b parsl.dataflow.memoization INFO: Task 2246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2246 try 0 launched on executor multi with executor id 2247 parsl.dataflow.dflow INFO: Standard output for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/48d980db-a79e-4e1e-b055-bc17b8edec2f_isrForCrosstalkSources_1031726_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/48d980db-a79e-4e1e-b055-bc17b8edec2f_isrForCrosstalkSources_1031726_4.stderr parsl.dataflow.dflow DEBUG: Task 2247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2247 has memoization hash dde3a7f569fc7c5083b4406cebe7c61a parsl.dataflow.memoization INFO: Task 2247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2247 try 0 launched on executor multi with executor id 2248 parsl.dataflow.dflow INFO: Standard output for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b37e545a-1e5c-4e91-a8c9-f0e87071c881_isrForCrosstalkSources_1031728_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b37e545a-1e5c-4e91-a8c9-f0e87071c881_isrForCrosstalkSources_1031728_41.stderr parsl.dataflow.dflow DEBUG: Task 2248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2248 has memoization hash f14d73bf14822bf1c874e32a2f8441c0 parsl.dataflow.memoization INFO: Task 2248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6920c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2248 try 0 launched on executor multi with executor id 2249 parsl.dataflow.dflow INFO: Standard output for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6b2e5859-3945-46e9-9d87-01a136f865df_isrForCrosstalkSources_1031726_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6b2e5859-3945-46e9-9d87-01a136f865df_isrForCrosstalkSources_1031726_39.stderr parsl.dataflow.dflow DEBUG: Task 2249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2249 has memoization hash 9c4535076425343ef1e35a4df3deafa2 parsl.dataflow.memoization INFO: Task 2249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2249 try 0 launched on executor multi with executor id 2250 parsl.dataflow.dflow INFO: Standard output for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a7347157-5d96-45e0-8a50-a7c962744e1e_isrForCrosstalkSources_1031731_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a7347157-5d96-45e0-8a50-a7c962744e1e_isrForCrosstalkSources_1031731_35.stderr parsl.dataflow.dflow DEBUG: Task 2250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2250 has memoization hash 9917acf32239ab52e75e5f7451664b50 parsl.dataflow.memoization INFO: Task 2250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6922a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2250 try 0 launched on executor multi with executor id 2251 parsl.dataflow.dflow INFO: Standard output for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4bbb9be2-3c46-47c1-9cde-4fe09bd27f31_isrForCrosstalkSources_1031709_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4bbb9be2-3c46-47c1-9cde-4fe09bd27f31_isrForCrosstalkSources_1031709_19.stderr parsl.dataflow.dflow DEBUG: Task 2251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2251 has memoization hash b00bb207a9d7a4f2d5db04be217518ee parsl.dataflow.memoization INFO: Task 2251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2251 try 0 launched on executor multi with executor id 2252 parsl.dataflow.dflow INFO: Standard output for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/701c56d9-68e2-4509-9b25-52ccd999f4e9_isrForCrosstalkSources_1031708_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/701c56d9-68e2-4509-9b25-52ccd999f4e9_isrForCrosstalkSources_1031708_60.stderr parsl.dataflow.dflow DEBUG: Task 2252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2252 has memoization hash 35560b5401eee12a12e818ed0989759c parsl.dataflow.memoization INFO: Task 2252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2252 try 0 launched on executor multi with executor id 2253 parsl.dataflow.dflow INFO: Standard output for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/4907fc1e-e455-456b-b62a-efa9500c91e2_isrForCrosstalkSources_1031726_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/4907fc1e-e455-456b-b62a-efa9500c91e2_isrForCrosstalkSources_1031726_53.stderr parsl.dataflow.dflow DEBUG: Task 2253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2253 has memoization hash 7b7a45c32d8b985e385c459424a68aa7 parsl.dataflow.memoization INFO: Task 2253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6923e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2253 try 0 launched on executor multi with executor id 2254 parsl.dataflow.dflow INFO: Standard output for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/6dccfee7-cfc7-4c5f-9c8c-8c4a61eeba3e_isrForCrosstalkSources_1031720_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/6dccfee7-cfc7-4c5f-9c8c-8c4a61eeba3e_isrForCrosstalkSources_1031720_55.stderr parsl.dataflow.dflow DEBUG: Task 2254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2254 has memoization hash 270373d679745aa907787803f5972966 parsl.dataflow.memoization INFO: Task 2254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2254 try 0 launched on executor multi with executor id 2255 parsl.dataflow.dflow INFO: Standard output for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/b68cf1f2-9be2-4a1f-94a3-1f8f1f13ac2c_isrForCrosstalkSources_1031733_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/b68cf1f2-9be2-4a1f-94a3-1f8f1f13ac2c_isrForCrosstalkSources_1031733_58.stderr parsl.dataflow.dflow DEBUG: Task 2255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2255 has memoization hash 68eff0c579b1410d1873058eed26c714 parsl.dataflow.memoization INFO: Task 2255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6925c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2255 try 0 launched on executor multi with executor id 2256 parsl.dataflow.dflow INFO: Standard output for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/975751e7-8e5f-45cf-b20d-67c7658ceeb8_isrForCrosstalkSources_1031712_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/975751e7-8e5f-45cf-b20d-67c7658ceeb8_isrForCrosstalkSources_1031712_3.stderr parsl.dataflow.dflow DEBUG: Task 2256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2256 has memoization hash 34631ab0b47aeeaae7ccd6ddae7e0c77 parsl.dataflow.memoization INFO: Task 2256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2256 try 0 launched on executor multi with executor id 2257 parsl.dataflow.dflow INFO: Standard output for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/61f6d2a0-fb57-428b-8021-8f0d0ceb88e8_isrForCrosstalkSources_1031718_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/61f6d2a0-fb57-428b-8021-8f0d0ceb88e8_isrForCrosstalkSources_1031718_5.stderr parsl.dataflow.dflow DEBUG: Task 2257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2257 has memoization hash 38269d6f49379d587b6517de2d5cb421 parsl.dataflow.memoization INFO: Task 2257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6927a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2257 try 0 launched on executor multi with executor id 2258 parsl.dataflow.dflow INFO: Standard output for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/773f1962-1ad9-4b8f-943a-3b6152d740c6_isrForCrosstalkSources_1031725_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/773f1962-1ad9-4b8f-943a-3b6152d740c6_isrForCrosstalkSources_1031725_27.stderr parsl.dataflow.dflow DEBUG: Task 2258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2258 has memoization hash 00a059f30f2cec20469935b5fc0c425a parsl.dataflow.memoization INFO: Task 2258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2258 try 0 launched on executor multi with executor id 2259 parsl.dataflow.dflow INFO: Standard output for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b584f179-1909-40b1-8bc8-2df4692617aa_isrForCrosstalkSources_1031707_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b584f179-1909-40b1-8bc8-2df4692617aa_isrForCrosstalkSources_1031707_33.stderr parsl.dataflow.dflow DEBUG: Task 2259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2259 has memoization hash b4dfb102450f92d6ee0c4ab0332ea9d9 parsl.dataflow.memoization INFO: Task 2259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2259 try 0 launched on executor multi with executor id 2260 parsl.dataflow.dflow INFO: Standard output for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/a26fc7f4-7404-4d64-873d-16eb6373521c_isrForCrosstalkSources_1031705_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/a26fc7f4-7404-4d64-873d-16eb6373521c_isrForCrosstalkSources_1031705_23.stderr parsl.dataflow.dflow DEBUG: Task 2260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2260 has memoization hash a5ce8cda040bc241f05816b16458b20e parsl.dataflow.memoization INFO: Task 2260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6928e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2260 try 0 launched on executor multi with executor id 2261 parsl.dataflow.dflow INFO: Standard output for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/996b35c8-e264-4486-ac26-03e177d3f90f_isrForCrosstalkSources_1031702_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/996b35c8-e264-4486-ac26-03e177d3f90f_isrForCrosstalkSources_1031702_30.stderr parsl.dataflow.dflow DEBUG: Task 2261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2261 has memoization hash 9109ff728abc51aa653ff7ab8187e6ec parsl.dataflow.memoization INFO: Task 2261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2261 try 0 launched on executor multi with executor id 2262 parsl.dataflow.dflow INFO: Standard output for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/729dd954-d477-42a7-98fd-f50cbf2b8812_isrForCrosstalkSources_1031727_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/729dd954-d477-42a7-98fd-f50cbf2b8812_isrForCrosstalkSources_1031727_21.stderr parsl.dataflow.dflow DEBUG: Task 2262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2262 has memoization hash 25267870a90cd6262044ad535b667879 parsl.dataflow.memoization INFO: Task 2262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2262 try 0 launched on executor multi with executor id 2263 parsl.dataflow.dflow INFO: Standard output for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/308f9690-e667-43a6-855d-cc3ca393802d_isrForCrosstalkSources_1031708_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/308f9690-e667-43a6-855d-cc3ca393802d_isrForCrosstalkSources_1031708_47.stderr parsl.dataflow.dflow DEBUG: Task 2263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2263 has memoization hash 8102bccb90f68a5a92585dd189d9d8e0 parsl.dataflow.memoization INFO: Task 2263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2263 try 0 launched on executor multi with executor id 2264 parsl.dataflow.dflow INFO: Standard output for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/83f0663f-2caf-48f4-9d6e-11ef305c15ab_isrForCrosstalkSources_1031721_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/83f0663f-2caf-48f4-9d6e-11ef305c15ab_isrForCrosstalkSources_1031721_28.stderr parsl.dataflow.dflow DEBUG: Task 2264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2264 has memoization hash 5463fc7b0f52a77b78a1d09cd8b7b313 parsl.dataflow.memoization INFO: Task 2264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2264 try 0 launched on executor multi with executor id 2265 parsl.dataflow.dflow INFO: Standard output for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2342d359-07c8-4594-9490-f6c0505e30cd_isrForCrosstalkSources_1031731_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2342d359-07c8-4594-9490-f6c0505e30cd_isrForCrosstalkSources_1031731_5.stderr parsl.dataflow.dflow DEBUG: Task 2265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2265 has memoization hash 9aeccc80986b751e7e6b8c29c615c28b parsl.dataflow.memoization INFO: Task 2265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2265 try 0 launched on executor multi with executor id 2266 parsl.dataflow.dflow INFO: Standard output for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6942f277-03e7-4e77-a0ed-a14bac9fc9a3_isrForCrosstalkSources_1031737_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6942f277-03e7-4e77-a0ed-a14bac9fc9a3_isrForCrosstalkSources_1031737_41.stderr parsl.dataflow.dflow DEBUG: Task 2266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2266 has memoization hash d6fc68401cd5bf295953c2a11b218107 parsl.dataflow.memoization INFO: Task 2266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2266 try 0 launched on executor multi with executor id 2267 parsl.dataflow.dflow INFO: Standard output for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a50a599f-7384-4290-a014-96a2f02508ce_isrForCrosstalkSources_1031711_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a50a599f-7384-4290-a014-96a2f02508ce_isrForCrosstalkSources_1031711_40.stderr parsl.dataflow.dflow DEBUG: Task 2267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2267 has memoization hash e34172efbf2a6819e6d3cd607f403946 parsl.dataflow.memoization INFO: Task 2267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2267 try 0 launched on executor multi with executor id 2268 parsl.dataflow.dflow INFO: Standard output for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/034a4f57-b09b-418e-a5e9-7164e4355108_isrForCrosstalkSources_1031718_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/034a4f57-b09b-418e-a5e9-7164e4355108_isrForCrosstalkSources_1031718_59.stderr parsl.dataflow.dflow DEBUG: Task 2268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2268 has memoization hash d45e525abbb51d2ec1938b654ab37787 parsl.dataflow.memoization INFO: Task 2268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2268 try 0 launched on executor multi with executor id 2269 parsl.dataflow.dflow INFO: Standard output for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0e7519fe-8016-4479-af2e-26eb37c8f9b2_isrForCrosstalkSources_1031706_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0e7519fe-8016-4479-af2e-26eb37c8f9b2_isrForCrosstalkSources_1031706_50.stderr parsl.dataflow.dflow DEBUG: Task 2269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2269 has memoization hash 9a703e5f172db4a06a296fe179da14be parsl.dataflow.memoization INFO: Task 2269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2269 try 0 launched on executor multi with executor id 2270 parsl.dataflow.dflow INFO: Standard output for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/09f57c58-32e2-4aab-99d2-d68aca0a8826_isrForCrosstalkSources_1031735_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/09f57c58-32e2-4aab-99d2-d68aca0a8826_isrForCrosstalkSources_1031735_2.stderr parsl.dataflow.dflow DEBUG: Task 2270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2270 has memoization hash e4e63631a1c9f27b056ed384f4b1a766 parsl.dataflow.memoization INFO: Task 2270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2270 try 0 launched on executor multi with executor id 2271 parsl.dataflow.dflow INFO: Standard output for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2f7678db-6ca9-4d10-b41c-e0efa83c34f5_isrForCrosstalkSources_1031708_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2f7678db-6ca9-4d10-b41c-e0efa83c34f5_isrForCrosstalkSources_1031708_22.stderr parsl.dataflow.dflow DEBUG: Task 2271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2271 has memoization hash 73e8170fc45a506aa41da418fa28a5b7 parsl.dataflow.memoization INFO: Task 2271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f692fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2271 try 0 launched on executor multi with executor id 2272 parsl.dataflow.dflow INFO: Standard output for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/86b7ba8a-232c-480d-bccf-61093d2d7cf8_isrForCrosstalkSources_1031703_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/86b7ba8a-232c-480d-bccf-61093d2d7cf8_isrForCrosstalkSources_1031703_43.stderr parsl.dataflow.dflow DEBUG: Task 2272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2272 has memoization hash 6833dee7bed69beabb96194a32d11478 parsl.dataflow.memoization INFO: Task 2272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2272 try 0 launched on executor multi with executor id 2273 parsl.dataflow.dflow INFO: Standard output for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/0b442013-f2bf-43f2-bf91-14b189e8bdf0_isrForCrosstalkSources_1031699_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/0b442013-f2bf-43f2-bf91-14b189e8bdf0_isrForCrosstalkSources_1031699_14.stderr parsl.dataflow.dflow DEBUG: Task 2273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2273 has memoization hash bd8fd2a3a05cba159a0556cca8298a6c parsl.dataflow.memoization INFO: Task 2273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2273 try 0 launched on executor multi with executor id 2274 parsl.dataflow.dflow INFO: Standard output for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3f0a5c2d-1541-42bb-b310-e52e4385e7a1_isrForCrosstalkSources_1031736_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3f0a5c2d-1541-42bb-b310-e52e4385e7a1_isrForCrosstalkSources_1031736_28.stderr parsl.dataflow.dflow DEBUG: Task 2274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2274 has memoization hash 44a67df91e7bc203914ea9244fd511ae parsl.dataflow.memoization INFO: Task 2274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6931a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2274 try 0 launched on executor multi with executor id 2275 parsl.dataflow.dflow INFO: Standard output for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a4ac7de7-aa2e-4afb-a843-9b87f2eee58a_isrForCrosstalkSources_1031719_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a4ac7de7-aa2e-4afb-a843-9b87f2eee58a_isrForCrosstalkSources_1031719_16.stderr parsl.dataflow.dflow DEBUG: Task 2275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2275 has memoization hash ce3f0547c9bdfe1f1853626533bf8218 parsl.dataflow.memoization INFO: Task 2275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2275 try 0 launched on executor multi with executor id 2276 parsl.dataflow.dflow INFO: Standard output for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/45b01147-3914-4346-9aa1-f450ff5f434e_isrForCrosstalkSources_1031702_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/45b01147-3914-4346-9aa1-f450ff5f434e_isrForCrosstalkSources_1031702_55.stderr parsl.dataflow.dflow DEBUG: Task 2276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2276 has memoization hash 7366f223aaf44149e83959fa72079cf9 parsl.dataflow.memoization INFO: Task 2276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6932e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2276 try 0 launched on executor multi with executor id 2277 parsl.dataflow.dflow INFO: Standard output for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8b364f63-a882-4cf8-9650-8c7fc0ab35af_isrForCrosstalkSources_1031733_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8b364f63-a882-4cf8-9650-8c7fc0ab35af_isrForCrosstalkSources_1031733_9.stderr parsl.dataflow.dflow DEBUG: Task 2277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2277 has memoization hash 1a430a1c5b12d81b31525f10124c84f3 parsl.dataflow.memoization INFO: Task 2277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2277 try 0 launched on executor multi with executor id 2278 parsl.dataflow.dflow INFO: Standard output for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c82c3b08-7fda-4e6b-82f4-8a4f6ab54a0b_isrForCrosstalkSources_1031719_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c82c3b08-7fda-4e6b-82f4-8a4f6ab54a0b_isrForCrosstalkSources_1031719_8.stderr parsl.dataflow.dflow DEBUG: Task 2278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2278 has memoization hash e22fa07c1692a76a6f445682dd5af29e parsl.dataflow.memoization INFO: Task 2278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2278 try 0 launched on executor multi with executor id 2279 parsl.dataflow.dflow INFO: Standard output for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/390aa2e0-2c71-46c2-ace2-0f72a90fc77a_isrForCrosstalkSources_1031705_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/390aa2e0-2c71-46c2-ace2-0f72a90fc77a_isrForCrosstalkSources_1031705_48.stderr parsl.dataflow.dflow DEBUG: Task 2279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2279 has memoization hash 5c2f642dd4e3b4f2553db6877c79fb54 parsl.dataflow.memoization INFO: Task 2279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6934c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2279 try 0 launched on executor multi with executor id 2280 parsl.dataflow.dflow INFO: Standard output for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/b164bfe6-9fb7-4bd0-9057-d3d3b3467efe_isrForCrosstalkSources_1031723_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/b164bfe6-9fb7-4bd0-9057-d3d3b3467efe_isrForCrosstalkSources_1031723_12.stderr parsl.dataflow.dflow DEBUG: Task 2280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2280 has memoization hash b9ab883a4163edfe23014e7972eae283 parsl.dataflow.memoization INFO: Task 2280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2280 try 0 launched on executor multi with executor id 2281 parsl.dataflow.dflow INFO: Standard output for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/16aa788b-6546-4014-8b03-2ff34d777d05_isrForCrosstalkSources_1031738_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/16aa788b-6546-4014-8b03-2ff34d777d05_isrForCrosstalkSources_1031738_7.stderr parsl.dataflow.dflow DEBUG: Task 2281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2281 has memoization hash d1bad6e70e0c016cdeb7242da25c5cc5 parsl.dataflow.memoization INFO: Task 2281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2281 try 0 launched on executor multi with executor id 2282 parsl.dataflow.dflow INFO: Standard output for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/fffc5bae-a33b-44a5-ac6c-c0fc23f7b84a_isrForCrosstalkSources_1031703_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/fffc5bae-a33b-44a5-ac6c-c0fc23f7b84a_isrForCrosstalkSources_1031703_30.stderr parsl.dataflow.dflow DEBUG: Task 2282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2282 has memoization hash 05dd89d22e9c830bb1277342762ff617 parsl.dataflow.memoization INFO: Task 2282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2282 try 0 launched on executor multi with executor id 2283 parsl.dataflow.dflow INFO: Standard output for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/ce6f2e3b-45fa-4620-819b-228fd7f20c81_isrForCrosstalkSources_1031714_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/ce6f2e3b-45fa-4620-819b-228fd7f20c81_isrForCrosstalkSources_1031714_58.stderr parsl.dataflow.dflow DEBUG: Task 2283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2283 has memoization hash b3c3783551476dd624061bbe50fe4d48 parsl.dataflow.memoization INFO: Task 2283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6936a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2283 try 0 launched on executor multi with executor id 2284 parsl.dataflow.dflow INFO: Standard output for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/0bd20edf-7a4f-44d8-9f7b-72767cdef6f2_isrForCrosstalkSources_1031731_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/0bd20edf-7a4f-44d8-9f7b-72767cdef6f2_isrForCrosstalkSources_1031731_6.stderr parsl.dataflow.dflow DEBUG: Task 2284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2284 has memoization hash fd588ee13920713a45e2c743c841d122 parsl.dataflow.memoization INFO: Task 2284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6937e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2284 try 0 launched on executor multi with executor id 2285 parsl.dataflow.dflow INFO: Standard output for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3920c937-b538-441f-8a1f-873625439cb2_isrForCrosstalkSources_1031729_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3920c937-b538-441f-8a1f-873625439cb2_isrForCrosstalkSources_1031729_46.stderr parsl.dataflow.dflow DEBUG: Task 2285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2285 has memoization hash 57e28d1048cb8fc3af25ba5f95507f21 parsl.dataflow.memoization INFO: Task 2285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2285 try 0 launched on executor multi with executor id 2286 parsl.dataflow.dflow INFO: Standard output for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/2a1b14a9-9389-4948-bf8e-fe4797a5336d_isrForCrosstalkSources_1031715_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/2a1b14a9-9389-4948-bf8e-fe4797a5336d_isrForCrosstalkSources_1031715_12.stderr parsl.dataflow.dflow DEBUG: Task 2286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2286 has memoization hash e1d287f90c63b600d1ec0082a0cbcb91 parsl.dataflow.memoization INFO: Task 2286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2286 try 0 launched on executor multi with executor id 2287 parsl.dataflow.dflow INFO: Standard output for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/918d01ea-740d-4e76-8819-6c590b8ce9e6_isrForCrosstalkSources_1031704_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/918d01ea-740d-4e76-8819-6c590b8ce9e6_isrForCrosstalkSources_1031704_22.stderr parsl.dataflow.dflow DEBUG: Task 2287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2287 has memoization hash f4d01b11c51681f0ea11561d52e0e518 parsl.dataflow.memoization INFO: Task 2287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2287 try 0 launched on executor multi with executor id 2288 parsl.dataflow.dflow INFO: Standard output for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/5073facb-268f-45a6-9f84-143a36f7b074_isrForCrosstalkSources_1031702_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/5073facb-268f-45a6-9f84-143a36f7b074_isrForCrosstalkSources_1031702_24.stderr parsl.dataflow.dflow DEBUG: Task 2288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2288 has memoization hash ee95811951f439e6f2e63751c8abdc94 parsl.dataflow.memoization INFO: Task 2288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f6939c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2288 try 0 launched on executor multi with executor id 2289 parsl.dataflow.dflow INFO: Standard output for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/4180daed-23d1-45b9-bdb0-a773180ecfe3_isrForCrosstalkSources_1031712_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/4180daed-23d1-45b9-bdb0-a773180ecfe3_isrForCrosstalkSources_1031712_33.stderr parsl.dataflow.dflow DEBUG: Task 2289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2289 has memoization hash 81114027e750d5f63bd0ca8051e39e94 parsl.dataflow.memoization INFO: Task 2289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2289 try 0 launched on executor multi with executor id 2290 parsl.dataflow.dflow INFO: Standard output for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c1ff2c72-596c-44d0-a5b8-6cf7061ff70f_isrForCrosstalkSources_1031719_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c1ff2c72-596c-44d0-a5b8-6cf7061ff70f_isrForCrosstalkSources_1031719_19.stderr parsl.dataflow.dflow DEBUG: Task 2290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2290 has memoization hash f3a2fe1be50a69f62e5ce06fb9f6c313 parsl.dataflow.memoization INFO: Task 2290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2290 try 0 launched on executor multi with executor id 2291 parsl.dataflow.dflow INFO: Standard output for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/0849d87b-5e2a-4be7-a5e1-e79bca64be8a_isrForCrosstalkSources_1031708_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/0849d87b-5e2a-4be7-a5e1-e79bca64be8a_isrForCrosstalkSources_1031708_34.stderr parsl.dataflow.dflow DEBUG: Task 2291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2291 has memoization hash 551ac8921b691b5d0d338b91b12456e4 parsl.dataflow.memoization INFO: Task 2291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2291 try 0 launched on executor multi with executor id 2292 parsl.dataflow.dflow INFO: Standard output for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/7c0e4fa1-52c0-4018-badf-b3f97585fcee_isrForCrosstalkSources_1031737_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/7c0e4fa1-52c0-4018-badf-b3f97585fcee_isrForCrosstalkSources_1031737_32.stderr parsl.dataflow.dflow DEBUG: Task 2292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2292 has memoization hash c9d9921049f25c71d87c27ad51894f2f parsl.dataflow.memoization INFO: Task 2292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2292 try 0 launched on executor multi with executor id 2293 parsl.dataflow.dflow INFO: Standard output for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/81225a3c-dff1-4b30-bbd2-f116db14c431_isrForCrosstalkSources_1031736_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/81225a3c-dff1-4b30-bbd2-f116db14c431_isrForCrosstalkSources_1031736_25.stderr parsl.dataflow.dflow DEBUG: Task 2293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2293 has memoization hash 42a7f9fff64e28c56355d80e3c7796c7 parsl.dataflow.memoization INFO: Task 2293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2293 try 0 launched on executor multi with executor id 2294 parsl.dataflow.dflow INFO: Standard output for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b5f34b82-4df3-44f5-a3f3-c8418b1ab601_isrForCrosstalkSources_1031713_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b5f34b82-4df3-44f5-a3f3-c8418b1ab601_isrForCrosstalkSources_1031713_16.stderr parsl.dataflow.dflow DEBUG: Task 2294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2294 has memoization hash eedc1ffb0e1bd1463dbeac8b240d4a08 parsl.dataflow.memoization INFO: Task 2294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2294 try 0 launched on executor multi with executor id 2295 parsl.dataflow.dflow INFO: Standard output for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/26c194a3-3990-4b85-a347-dd8d49ca41a5_isrForCrosstalkSources_1031726_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/26c194a3-3990-4b85-a347-dd8d49ca41a5_isrForCrosstalkSources_1031726_29.stderr parsl.dataflow.dflow DEBUG: Task 2295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2295 has memoization hash d5885ee2ca971febac9474b5bd268332 parsl.dataflow.memoization INFO: Task 2295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2295 try 0 launched on executor multi with executor id 2296 parsl.dataflow.dflow INFO: Standard output for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/d16902bb-1586-4fa4-b92a-2c387d862085_isrForCrosstalkSources_1031708_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/d16902bb-1586-4fa4-b92a-2c387d862085_isrForCrosstalkSources_1031708_21.stderr parsl.dataflow.dflow DEBUG: Task 2296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2296 has memoization hash 75c212660c5907b0b46f85a1450304bd parsl.dataflow.memoization INFO: Task 2296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f693f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2296 try 0 launched on executor multi with executor id 2297 parsl.dataflow.dflow INFO: Standard output for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e3b7f9f8-1312-4b97-a199-5ecd6a6660ec_isrForCrosstalkSources_1031710_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e3b7f9f8-1312-4b97-a199-5ecd6a6660ec_isrForCrosstalkSources_1031710_53.stderr parsl.dataflow.dflow DEBUG: Task 2297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2297 has memoization hash b64bdf54906d60b00bce9ba64bc878e7 parsl.dataflow.memoization INFO: Task 2297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f690040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2297 try 0 launched on executor multi with executor id 2298 parsl.dataflow.dflow INFO: Standard output for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/28b29384-decf-4acd-8542-ea0f4f91c76f_isrForCrosstalkSources_1031713_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/28b29384-decf-4acd-8542-ea0f4f91c76f_isrForCrosstalkSources_1031713_39.stderr parsl.dataflow.dflow DEBUG: Task 2298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2298 has memoization hash 26f405ba4ad7d5314bdb4e5e2ddd4c34 parsl.dataflow.memoization INFO: Task 2298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2298 try 0 launched on executor multi with executor id 2299 parsl.dataflow.dflow INFO: Standard output for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/c52f9ccd-13a2-4dcc-b50c-2c6077be4f54_isrForCrosstalkSources_1031699_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/c52f9ccd-13a2-4dcc-b50c-2c6077be4f54_isrForCrosstalkSources_1031699_41.stderr parsl.dataflow.dflow DEBUG: Task 2299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2299 has memoization hash 7ad800c4c0f904c516d83470520a4dbb parsl.dataflow.memoization INFO: Task 2299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2299 try 0 launched on executor multi with executor id 2300 parsl.dataflow.dflow INFO: Standard output for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/3931b165-a5b9-4f96-9222-d847bd5d812a_isrForCrosstalkSources_1031718_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/3931b165-a5b9-4f96-9222-d847bd5d812a_isrForCrosstalkSources_1031718_8.stderr parsl.dataflow.dflow DEBUG: Task 2300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2300 has memoization hash 8be1ecee1546b5a5e3b1923a91b93fff parsl.dataflow.memoization INFO: Task 2300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2300 try 0 launched on executor multi with executor id 2301 parsl.dataflow.dflow INFO: Standard output for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d7b7ffdd-da45-442a-91ba-1b17720e7c64_isrForCrosstalkSources_1031725_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d7b7ffdd-da45-442a-91ba-1b17720e7c64_isrForCrosstalkSources_1031725_54.stderr parsl.dataflow.dflow DEBUG: Task 2301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2301 has memoization hash d855e37aebeecbf764813d7645554ddf parsl.dataflow.memoization INFO: Task 2301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5342c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2301 try 0 launched on executor multi with executor id 2302 parsl.dataflow.dflow INFO: Standard output for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/ebf3df0d-23ba-4f35-8a38-064ed7596412_isrForCrosstalkSources_1031720_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/ebf3df0d-23ba-4f35-8a38-064ed7596412_isrForCrosstalkSources_1031720_41.stderr parsl.dataflow.dflow DEBUG: Task 2302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2302 has memoization hash ffaea3e6c45d316feec731a8d41dc971 parsl.dataflow.memoization INFO: Task 2302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2302 try 0 launched on executor multi with executor id 2303 parsl.dataflow.dflow INFO: Standard output for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3b475d43-3300-4c13-9a79-a74d23b53d4e_isrForCrosstalkSources_1031723_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3b475d43-3300-4c13-9a79-a74d23b53d4e_isrForCrosstalkSources_1031723_8.stderr parsl.dataflow.dflow DEBUG: Task 2303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2303 has memoization hash 735c56c72887b15f70622a0733c09b2a parsl.dataflow.memoization INFO: Task 2303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2303 try 0 launched on executor multi with executor id 2304 parsl.dataflow.dflow INFO: Standard output for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/da7d8998-e87e-4f54-9c27-434f14e3a322_isrForCrosstalkSources_1031704_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/da7d8998-e87e-4f54-9c27-434f14e3a322_isrForCrosstalkSources_1031704_44.stderr parsl.dataflow.dflow DEBUG: Task 2304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2304 has memoization hash 1c941b669d2adbfb27cb7366c510e238 parsl.dataflow.memoization INFO: Task 2304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5344a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2304 try 0 launched on executor multi with executor id 2305 parsl.dataflow.dflow INFO: Standard output for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/89a68929-d11f-4b65-961d-fa28381982ae_isrForCrosstalkSources_1031737_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/89a68929-d11f-4b65-961d-fa28381982ae_isrForCrosstalkSources_1031737_14.stderr parsl.dataflow.dflow DEBUG: Task 2305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2305 has memoization hash f451b9dbe7ad9f7cb6f867beb97b358e parsl.dataflow.memoization INFO: Task 2305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2305 try 0 launched on executor multi with executor id 2306 parsl.dataflow.dflow INFO: Standard output for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/a200cf15-86da-4dbc-ba08-a825d4654fef_isrForCrosstalkSources_1031703_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/a200cf15-86da-4dbc-ba08-a825d4654fef_isrForCrosstalkSources_1031703_15.stderr parsl.dataflow.dflow DEBUG: Task 2306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2306 has memoization hash ae3f1472067103ed6a5c1d4b09cc0c13 parsl.dataflow.memoization INFO: Task 2306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5345e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2306 try 0 launched on executor multi with executor id 2307 parsl.dataflow.dflow INFO: Standard output for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/513208a5-6715-4e36-b5aa-f16ab42bec37_isrForCrosstalkSources_1031726_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/513208a5-6715-4e36-b5aa-f16ab42bec37_isrForCrosstalkSources_1031726_9.stderr parsl.dataflow.dflow DEBUG: Task 2307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2307 has memoization hash 4a01e1679734d0fa944652e6aeb3009b parsl.dataflow.memoization INFO: Task 2307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2307 try 0 launched on executor multi with executor id 2308 parsl.dataflow.dflow INFO: Standard output for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/80e84c44-4210-4318-8f9e-96ab28d6a51e_isrForCrosstalkSources_1031720_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/80e84c44-4210-4318-8f9e-96ab28d6a51e_isrForCrosstalkSources_1031720_15.stderr parsl.dataflow.dflow DEBUG: Task 2308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2308 has memoization hash 1fdf6756414bbe6ac6fbc111c847da3b parsl.dataflow.memoization INFO: Task 2308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2308 try 0 launched on executor multi with executor id 2309 parsl.dataflow.dflow INFO: Standard output for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2a94c002-6195-4c43-9903-33005cfffbde_isrForCrosstalkSources_1031736_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2a94c002-6195-4c43-9903-33005cfffbde_isrForCrosstalkSources_1031736_55.stderr parsl.dataflow.dflow DEBUG: Task 2309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2309 has memoization hash 0577ffced6258f5a31fec88b6f711ea5 parsl.dataflow.memoization INFO: Task 2309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5347c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2309 try 0 launched on executor multi with executor id 2310 parsl.dataflow.dflow INFO: Standard output for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/dd384ccd-bdaa-410c-a352-2f456911556a_isrForCrosstalkSources_1031731_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/dd384ccd-bdaa-410c-a352-2f456911556a_isrForCrosstalkSources_1031731_8.stderr parsl.dataflow.dflow DEBUG: Task 2310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2310 has memoization hash cb0d573b6a3804e7ac738e4bdaf3c133 parsl.dataflow.memoization INFO: Task 2310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2310 try 0 launched on executor multi with executor id 2311 parsl.dataflow.dflow INFO: Standard output for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9d2d0e35-bc6b-484b-a5b8-92a84d681b88_isrForCrosstalkSources_1031733_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9d2d0e35-bc6b-484b-a5b8-92a84d681b88_isrForCrosstalkSources_1031733_47.stderr parsl.dataflow.dflow DEBUG: Task 2311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2311 has memoization hash 0439b2f15880d4c712237bbddb1e44cc parsl.dataflow.memoization INFO: Task 2311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2311 try 0 launched on executor multi with executor id 2312 parsl.dataflow.dflow INFO: Standard output for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/8f1ab18e-26a6-4303-a0c2-622d79f1a5cd_isrForCrosstalkSources_1031702_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/8f1ab18e-26a6-4303-a0c2-622d79f1a5cd_isrForCrosstalkSources_1031702_54.stderr parsl.dataflow.dflow DEBUG: Task 2312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2312 has memoization hash 80fe589b0f1938dd836921d73c059f16 parsl.dataflow.memoization INFO: Task 2312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2312 try 0 launched on executor multi with executor id 2313 parsl.dataflow.dflow INFO: Standard output for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3aa9694f-450a-4c24-ab94-635bd2e87cda_isrForCrosstalkSources_1031707_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3aa9694f-450a-4c24-ab94-635bd2e87cda_isrForCrosstalkSources_1031707_32.stderr parsl.dataflow.dflow DEBUG: Task 2313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2313 has memoization hash efe3df1f2e1b10e6accdd21efd0392b5 parsl.dataflow.memoization INFO: Task 2313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5349a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2313 try 0 launched on executor multi with executor id 2314 parsl.dataflow.dflow INFO: Standard output for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/609a9c14-39e5-4ae2-86b2-6e9a78b5a6e2_isrForCrosstalkSources_1031713_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/609a9c14-39e5-4ae2-86b2-6e9a78b5a6e2_isrForCrosstalkSources_1031713_44.stderr parsl.dataflow.dflow DEBUG: Task 2314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2314 has memoization hash 25d299fdd4a07fd6e7126b0460dfd1d7 parsl.dataflow.memoization INFO: Task 2314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2314 try 0 launched on executor multi with executor id 2315 parsl.dataflow.dflow INFO: Standard output for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b896192f-406a-4e27-bfdc-077362d78ca8_isrForCrosstalkSources_1031718_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b896192f-406a-4e27-bfdc-077362d78ca8_isrForCrosstalkSources_1031718_51.stderr parsl.dataflow.dflow DEBUG: Task 2315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2315 has memoization hash 3d766ae3f150a3dea83ce1106f7a17d2 parsl.dataflow.memoization INFO: Task 2315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2315 try 0 launched on executor multi with executor id 2316 parsl.dataflow.dflow INFO: Standard output for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/9969de03-e9c4-4996-a09e-3801d0b9513a_isrForCrosstalkSources_1031706_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/9969de03-e9c4-4996-a09e-3801d0b9513a_isrForCrosstalkSources_1031706_12.stderr parsl.dataflow.dflow DEBUG: Task 2316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2316 has memoization hash ed1858ea1fdec3235e5a05fa84820ba8 parsl.dataflow.memoization INFO: Task 2316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2316 try 0 launched on executor multi with executor id 2317 parsl.dataflow.dflow INFO: Standard output for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/cb3231f2-6f79-4338-a680-d667d44fa422_isrForCrosstalkSources_1031729_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/cb3231f2-6f79-4338-a680-d667d44fa422_isrForCrosstalkSources_1031729_37.stderr parsl.dataflow.dflow DEBUG: Task 2317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2317 has memoization hash 75c32fb73eff1b5da5dd16df1a747e4b parsl.dataflow.memoization INFO: Task 2317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 04:02:24 proc_lsst.multi:170 [INFO] [multi] job_id 17147958 proc_lsst.multi INFO: [multi] job_id 17147958 2024-03-17 04:02:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-17 04:02:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147958 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147958 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17147958 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-17 04:02:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-17 04:02:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-17 04:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2317 try 0 launched on executor multi with executor id 2318 parsl.dataflow.dflow INFO: Standard output for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/2f3e982f-0df0-4cf2-ab6f-4c2f735d9679_isrForCrosstalkSources_1031712_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/2f3e982f-0df0-4cf2-ab6f-4c2f735d9679_isrForCrosstalkSources_1031712_41.stderr parsl.dataflow.dflow DEBUG: Task 2318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2318 has memoization hash 3e326489dfd6fe73d2f89f4ae82f16fb parsl.dataflow.memoization INFO: Task 2318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2318 try 0 launched on executor multi with executor id 2319 parsl.dataflow.dflow INFO: Standard output for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c8b700b1-d645-4fea-b84b-7d9617745e96_isrForCrosstalkSources_1031710_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c8b700b1-d645-4fea-b84b-7d9617745e96_isrForCrosstalkSources_1031710_22.stderr parsl.dataflow.dflow DEBUG: Task 2319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2319 has memoization hash 3ac058a4ea6824ef164ed6111642edee parsl.dataflow.memoization INFO: Task 2319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f607740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2319 try 0 launched on executor multi with executor id 2320 parsl.dataflow.dflow INFO: Standard output for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/91ebfe14-9b52-4adb-ab28-bf6a5a75271d_isrForCrosstalkSources_1031699_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/91ebfe14-9b52-4adb-ab28-bf6a5a75271d_isrForCrosstalkSources_1031699_43.stderr parsl.dataflow.dflow DEBUG: Task 2320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2320 has memoization hash c760cc60aec90cff885c52b60174c52d parsl.dataflow.memoization INFO: Task 2320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f606200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2320 try 0 launched on executor multi with executor id 2321 parsl.dataflow.dflow INFO: Standard output for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4df7ae11-b047-4391-a3ab-6c3100364185_isrForCrosstalkSources_1031723_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4df7ae11-b047-4391-a3ab-6c3100364185_isrForCrosstalkSources_1031723_46.stderr parsl.dataflow.dflow DEBUG: Task 2321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2321 has memoization hash 25d5413aa1d89283e16f894e36006bf2 parsl.dataflow.memoization INFO: Task 2321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2321 try 0 launched on executor multi with executor id 2322 parsl.dataflow.dflow INFO: Standard output for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/1b5a87fb-20c4-4baa-ab7c-3742a995a0cc_isrForCrosstalkSources_1031728_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/1b5a87fb-20c4-4baa-ab7c-3742a995a0cc_isrForCrosstalkSources_1031728_36.stderr parsl.dataflow.dflow DEBUG: Task 2322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2322 has memoization hash 287d8f90c8ff6ff37db8a84d5766907f parsl.dataflow.memoization INFO: Task 2322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2322 try 0 launched on executor multi with executor id 2323 parsl.dataflow.dflow INFO: Standard output for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e97e4c7d-4354-41c3-a81f-f21927836fd2_isrForCrosstalkSources_1031725_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e97e4c7d-4354-41c3-a81f-f21927836fd2_isrForCrosstalkSources_1031725_16.stderr parsl.dataflow.dflow DEBUG: Task 2323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2323 has memoization hash 74d34547f9c79ce62561cd250772a6b7 parsl.dataflow.memoization INFO: Task 2323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2fc47ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2323 try 0 launched on executor multi with executor id 2324 parsl.dataflow.dflow INFO: Standard output for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/203960ac-d058-4239-8e11-49b0046d2c5c_isrForCrosstalkSources_1031714_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/203960ac-d058-4239-8e11-49b0046d2c5c_isrForCrosstalkSources_1031714_54.stderr parsl.dataflow.dflow DEBUG: Task 2324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2324 has memoization hash 52920f29d09b2b097eb56232437005d3 parsl.dataflow.memoization INFO: Task 2324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2324 try 0 launched on executor multi with executor id 2325 parsl.dataflow.dflow INFO: Standard output for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4354122d-1a4e-462c-be23-c02b1b6eacba_isrForCrosstalkSources_1031715_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4354122d-1a4e-462c-be23-c02b1b6eacba_isrForCrosstalkSources_1031715_23.stderr parsl.dataflow.dflow DEBUG: Task 2325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2325 has memoization hash 01c60726c9a161c912b555a244aa811f parsl.dataflow.memoization INFO: Task 2325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5340e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2325 try 0 launched on executor multi with executor id 2326 parsl.dataflow.dflow INFO: Standard output for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3bd4be64-d344-47b4-b157-864927ed407c_isrForCrosstalkSources_1031707_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3bd4be64-d344-47b4-b157-864927ed407c_isrForCrosstalkSources_1031707_18.stderr parsl.dataflow.dflow DEBUG: Task 2326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2326 has memoization hash deac9a0acd1119f3575fdb6b3c695d73 parsl.dataflow.memoization INFO: Task 2326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2326 try 0 launched on executor multi with executor id 2327 parsl.dataflow.dflow INFO: Standard output for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/87e9875b-ced8-4266-89fb-6f4ba77d0add_isrForCrosstalkSources_1031715_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/87e9875b-ced8-4266-89fb-6f4ba77d0add_isrForCrosstalkSources_1031715_16.stderr parsl.dataflow.dflow DEBUG: Task 2327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2327 has memoization hash 2440fe41db591f919eb7c2a1fa933fdf parsl.dataflow.memoization INFO: Task 2327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2327 try 0 launched on executor multi with executor id 2328 parsl.dataflow.dflow INFO: Standard output for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0cfff05e-c31d-4780-b1f6-65fae78a022c_isrForCrosstalkSources_1031712_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0cfff05e-c31d-4780-b1f6-65fae78a022c_isrForCrosstalkSources_1031712_20.stderr parsl.dataflow.dflow DEBUG: Task 2328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2328 has memoization hash ad4a7e2eb95a188e70c1ea375bab27a3 parsl.dataflow.memoization INFO: Task 2328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2328 try 0 launched on executor multi with executor id 2329 parsl.dataflow.dflow INFO: Standard output for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/62c38495-baf8-44dc-92b1-2def91889736_isrForCrosstalkSources_1031718_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/62c38495-baf8-44dc-92b1-2def91889736_isrForCrosstalkSources_1031718_22.stderr parsl.dataflow.dflow DEBUG: Task 2329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2329 has memoization hash 996e1fe310b4950e19b6d0c442679c73 parsl.dataflow.memoization INFO: Task 2329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2329 try 0 launched on executor multi with executor id 2330 parsl.dataflow.dflow INFO: Standard output for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ac0f6e5a-e758-4f74-8601-3bc6ff65026b_isrForCrosstalkSources_1031731_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ac0f6e5a-e758-4f74-8601-3bc6ff65026b_isrForCrosstalkSources_1031731_54.stderr parsl.dataflow.dflow DEBUG: Task 2330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2330 has memoization hash 8b86ba0dba427a5850cf99a974ea617b parsl.dataflow.memoization INFO: Task 2330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2330 try 0 launched on executor multi with executor id 2331 parsl.dataflow.dflow INFO: Standard output for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a74ca528-ddc2-4773-982d-65bb1fcd4276_isrForCrosstalkSources_1031719_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a74ca528-ddc2-4773-982d-65bb1fcd4276_isrForCrosstalkSources_1031719_25.stderr parsl.dataflow.dflow DEBUG: Task 2331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2331 has memoization hash ea375a0dfe476d949481007f5db6e0e9 parsl.dataflow.memoization INFO: Task 2331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2331 try 0 launched on executor multi with executor id 2332 parsl.dataflow.dflow INFO: Standard output for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/50ff80b2-a89a-4c4f-902f-1d8bd3075bad_isrForCrosstalkSources_1031709_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/50ff80b2-a89a-4c4f-902f-1d8bd3075bad_isrForCrosstalkSources_1031709_18.stderr parsl.dataflow.dflow DEBUG: Task 2332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2332 has memoization hash d8e74acb124d9308edbc78dfd4fe9303 parsl.dataflow.memoization INFO: Task 2332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5353a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2332 try 0 launched on executor multi with executor id 2333 parsl.dataflow.dflow INFO: Standard output for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/03b37c98-ab10-4d02-bda9-7839f8771626_isrForCrosstalkSources_1031707_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/03b37c98-ab10-4d02-bda9-7839f8771626_isrForCrosstalkSources_1031707_51.stderr parsl.dataflow.dflow DEBUG: Task 2333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2333 has memoization hash 37313bf6ed02803d03cc5f08e52feefd parsl.dataflow.memoization INFO: Task 2333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2333 try 0 launched on executor multi with executor id 2334 parsl.dataflow.dflow INFO: Standard output for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cfbce4b6-fed2-4a5d-ad2a-89bf6677ab1f_isrForCrosstalkSources_1031716_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cfbce4b6-fed2-4a5d-ad2a-89bf6677ab1f_isrForCrosstalkSources_1031716_19.stderr parsl.dataflow.dflow DEBUG: Task 2334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2334 has memoization hash 4b6f999c5aca3cae4fa862cf2d4c46a3 parsl.dataflow.memoization INFO: Task 2334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2334 try 0 launched on executor multi with executor id 2335 parsl.dataflow.dflow INFO: Standard output for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/76ded7eb-e838-4816-83d0-8c7f3f159d70_isrForCrosstalkSources_1031721_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/76ded7eb-e838-4816-83d0-8c7f3f159d70_isrForCrosstalkSources_1031721_59.stderr parsl.dataflow.dflow DEBUG: Task 2335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2335 has memoization hash a91b0e8c7adb0775a8cad696cba51528 parsl.dataflow.memoization INFO: Task 2335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5354e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2335 try 0 launched on executor multi with executor id 2336 parsl.dataflow.dflow INFO: Standard output for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/22e43c29-ac28-476e-acc6-b28ca2d062c5_isrForCrosstalkSources_1031704_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/22e43c29-ac28-476e-acc6-b28ca2d062c5_isrForCrosstalkSources_1031704_18.stderr parsl.dataflow.dflow DEBUG: Task 2336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2336 has memoization hash 53588846d7ab8d4f0b1ce586732ee653 parsl.dataflow.memoization INFO: Task 2336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2336 try 0 launched on executor multi with executor id 2337 parsl.dataflow.dflow INFO: Standard output for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d19efe44-41c3-4348-ac03-730aff6463cb_isrForCrosstalkSources_1031734_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d19efe44-41c3-4348-ac03-730aff6463cb_isrForCrosstalkSources_1031734_24.stderr parsl.dataflow.dflow DEBUG: Task 2337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2337 has memoization hash d61655169242869dd268caa99fd3f05a parsl.dataflow.memoization INFO: Task 2337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2337 try 0 launched on executor multi with executor id 2338 parsl.dataflow.dflow INFO: Standard output for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/1ad69b10-7057-4641-88fb-90ac02daec38_isrForCrosstalkSources_1031722_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/1ad69b10-7057-4641-88fb-90ac02daec38_isrForCrosstalkSources_1031722_53.stderr parsl.dataflow.dflow DEBUG: Task 2338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2338 has memoization hash 83aff81d0cf8543d7437e82170c761dd parsl.dataflow.memoization INFO: Task 2338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2338 try 0 launched on executor multi with executor id 2339 parsl.dataflow.dflow INFO: Standard output for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/16bb19ec-6b9f-427c-8e22-4774cb9d6a62_isrForCrosstalkSources_1031723_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/16bb19ec-6b9f-427c-8e22-4774cb9d6a62_isrForCrosstalkSources_1031723_31.stderr parsl.dataflow.dflow DEBUG: Task 2339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2339 has memoization hash 4a10c0824d104eef31f9669e99d6b323 parsl.dataflow.memoization INFO: Task 2339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2339 try 0 launched on executor multi with executor id 2340 parsl.dataflow.dflow INFO: Standard output for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e5cb5e25-a91a-488d-9647-4feb345f39a7_isrForCrosstalkSources_1031703_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e5cb5e25-a91a-488d-9647-4feb345f39a7_isrForCrosstalkSources_1031703_14.stderr parsl.dataflow.dflow DEBUG: Task 2340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2340 has memoization hash 997d8078161119baa2dabb8fe045e2fd parsl.dataflow.memoization INFO: Task 2340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2340 try 0 launched on executor multi with executor id 2341 parsl.dataflow.dflow INFO: Standard output for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/6d1af999-7507-4d6d-be5c-de1174fcb2c3_isrForCrosstalkSources_1031731_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/6d1af999-7507-4d6d-be5c-de1174fcb2c3_isrForCrosstalkSources_1031731_21.stderr parsl.dataflow.dflow DEBUG: Task 2341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2341 has memoization hash 4bbbcb3bd0a3f53df214fd7e67597acf parsl.dataflow.memoization INFO: Task 2341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5358a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2341 try 0 launched on executor multi with executor id 2342 parsl.dataflow.dflow INFO: Standard output for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2d1be4c6-f1f3-4215-99cf-22a0d8a62d97_isrForCrosstalkSources_1031726_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2d1be4c6-f1f3-4215-99cf-22a0d8a62d97_isrForCrosstalkSources_1031726_20.stderr parsl.dataflow.dflow DEBUG: Task 2342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2342 has memoization hash 121101f70b93bdbf9f1b247f3a9412c1 parsl.dataflow.memoization INFO: Task 2342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2342 try 0 launched on executor multi with executor id 2343 parsl.dataflow.dflow INFO: Standard output for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f55062d8-60bb-437d-b2a2-76525e77a227_isrForCrosstalkSources_1031716_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f55062d8-60bb-437d-b2a2-76525e77a227_isrForCrosstalkSources_1031716_30.stderr parsl.dataflow.dflow DEBUG: Task 2343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2343 has memoization hash 5ac09df2ed9ee1c711dfffbef02939ce parsl.dataflow.memoization INFO: Task 2343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2343 try 0 launched on executor multi with executor id 2344 parsl.dataflow.dflow INFO: Standard output for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/6377f5b2-aab4-4dc6-ba03-212c79047005_isrForCrosstalkSources_1031708_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/6377f5b2-aab4-4dc6-ba03-212c79047005_isrForCrosstalkSources_1031708_52.stderr parsl.dataflow.dflow DEBUG: Task 2344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2344 has memoization hash 42b5cf05a5521be1652c8aa8bae83cde parsl.dataflow.memoization INFO: Task 2344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2344 try 0 launched on executor multi with executor id 2345 parsl.dataflow.dflow INFO: Standard output for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/33d8cfc1-1496-4320-a0c3-eb0b00a78542_isrForCrosstalkSources_1031721_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/33d8cfc1-1496-4320-a0c3-eb0b00a78542_isrForCrosstalkSources_1031721_30.stderr parsl.dataflow.dflow DEBUG: Task 2345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2345 has memoization hash 67cd83e6de6be6f6df748df446f88c8b parsl.dataflow.memoization INFO: Task 2345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2345 try 0 launched on executor multi with executor id 2346 parsl.dataflow.dflow INFO: Standard output for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c2049d95-b2d7-4ca7-b614-173605e058ee_isrForCrosstalkSources_1031727_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c2049d95-b2d7-4ca7-b614-173605e058ee_isrForCrosstalkSources_1031727_20.stderr parsl.dataflow.dflow DEBUG: Task 2346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2346 has memoization hash 368af57b16c2043837703512f8748062 parsl.dataflow.memoization INFO: Task 2346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2346 try 0 launched on executor multi with executor id 2347 parsl.dataflow.dflow INFO: Standard output for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/9e89c959-39b4-4578-90af-13d831325270_isrForCrosstalkSources_1031722_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/9e89c959-39b4-4578-90af-13d831325270_isrForCrosstalkSources_1031722_28.stderr parsl.dataflow.dflow DEBUG: Task 2347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2347 has memoization hash 66b723a4535c638d1d3155b52078411a parsl.dataflow.memoization INFO: Task 2347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2347 try 0 launched on executor multi with executor id 2348 parsl.dataflow.dflow INFO: Standard output for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/19159d90-4f13-47d4-8c87-5af963cddfb4_isrForCrosstalkSources_1031704_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/19159d90-4f13-47d4-8c87-5af963cddfb4_isrForCrosstalkSources_1031704_52.stderr parsl.dataflow.dflow DEBUG: Task 2348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2348 has memoization hash 8aa39df45a99b72a57e22e3a13dfd307 parsl.dataflow.memoization INFO: Task 2348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2348 try 0 launched on executor multi with executor id 2349 parsl.dataflow.dflow INFO: Standard output for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ed4208fd-0b5c-4aa9-a6c5-62315321e8d8_isrForCrosstalkSources_1031699_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ed4208fd-0b5c-4aa9-a6c5-62315321e8d8_isrForCrosstalkSources_1031699_24.stderr parsl.dataflow.dflow DEBUG: Task 2349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2349 has memoization hash e88b6d4f23d1c46c0af9e8d2556ee43d parsl.dataflow.memoization INFO: Task 2349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2349 try 0 launched on executor multi with executor id 2350 parsl.dataflow.dflow INFO: Standard output for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/6a853279-2f01-456b-a18e-8eb8806552f0_isrForCrosstalkSources_1031710_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/6a853279-2f01-456b-a18e-8eb8806552f0_isrForCrosstalkSources_1031710_23.stderr parsl.dataflow.dflow DEBUG: Task 2350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2350 has memoization hash 2a53587378eec7577dc1e2407ea8a56f parsl.dataflow.memoization INFO: Task 2350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2350 try 0 launched on executor multi with executor id 2351 parsl.dataflow.dflow INFO: Standard output for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8c1ae541-3c12-4459-9002-cbd8be159b7a_isrForCrosstalkSources_1031709_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8c1ae541-3c12-4459-9002-cbd8be159b7a_isrForCrosstalkSources_1031709_47.stderr parsl.dataflow.dflow DEBUG: Task 2351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2351 has memoization hash a624d34c2943b7c8cb45901491bb1556 parsl.dataflow.memoization INFO: Task 2351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2351 try 0 launched on executor multi with executor id 2352 parsl.dataflow.dflow INFO: Standard output for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/7819bb82-2a37-4728-9b10-0c2b3885650f_isrForCrosstalkSources_1031720_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/7819bb82-2a37-4728-9b10-0c2b3885650f_isrForCrosstalkSources_1031720_45.stderr parsl.dataflow.dflow DEBUG: Task 2352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2352 has memoization hash dda15c53d29e3d346ae4f9856bc157c5 parsl.dataflow.memoization INFO: Task 2352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f535da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2352 try 0 launched on executor multi with executor id 2353 parsl.dataflow.dflow INFO: Standard output for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5a6a6b87-d595-4bca-8ff1-82e9f75e2007_isrForCrosstalkSources_1031705_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5a6a6b87-d595-4bca-8ff1-82e9f75e2007_isrForCrosstalkSources_1031705_24.stderr parsl.dataflow.dflow DEBUG: Task 2353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2353 has memoization hash c488847cc601d9e7c5f9d5f0ee5c902e parsl.dataflow.memoization INFO: Task 2353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2353 try 0 launched on executor multi with executor id 2354 parsl.dataflow.dflow INFO: Standard output for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e7bf91ec-e0d5-48aa-9e6f-589b27c0d8e8_isrForCrosstalkSources_1031725_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e7bf91ec-e0d5-48aa-9e6f-589b27c0d8e8_isrForCrosstalkSources_1031725_1.stderr parsl.dataflow.dflow DEBUG: Task 2354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2354 has memoization hash 86fe0fcbc24ebb19dba0f75a3ba5f2f5 parsl.dataflow.memoization INFO: Task 2354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5360c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2354 try 0 launched on executor multi with executor id 2355 parsl.dataflow.dflow INFO: Standard output for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fc9434b7-ab14-4d2b-a985-bf5a9b4a94e2_isrForCrosstalkSources_1031702_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fc9434b7-ab14-4d2b-a985-bf5a9b4a94e2_isrForCrosstalkSources_1031702_39.stderr parsl.dataflow.dflow DEBUG: Task 2355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2355 has memoization hash 57de53f76c8fc562d18c7ee5e21dbfd1 parsl.dataflow.memoization INFO: Task 2355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2355 try 0 launched on executor multi with executor id 2356 parsl.dataflow.dflow INFO: Standard output for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/418c6006-20c2-4203-9125-52cb1d04ce56_isrForCrosstalkSources_1031701_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/418c6006-20c2-4203-9125-52cb1d04ce56_isrForCrosstalkSources_1031701_57.stderr parsl.dataflow.dflow DEBUG: Task 2356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2356 has memoization hash 9bd290e34d45544847929bda6b92aed4 parsl.dataflow.memoization INFO: Task 2356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5362a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2356 try 0 launched on executor multi with executor id 2357 parsl.dataflow.dflow INFO: Standard output for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/12e18dcf-b01d-4c01-93f4-d0516ef2be75_isrForCrosstalkSources_1031712_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/12e18dcf-b01d-4c01-93f4-d0516ef2be75_isrForCrosstalkSources_1031712_52.stderr parsl.dataflow.dflow DEBUG: Task 2357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2357 has memoization hash 30d0a6c4b9e2b8350adaee6ffb514943 parsl.dataflow.memoization INFO: Task 2357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2357 try 0 launched on executor multi with executor id 2358 parsl.dataflow.dflow INFO: Standard output for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4a045573-bb18-436c-84a4-b2b40e966312_isrForCrosstalkSources_1031720_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4a045573-bb18-436c-84a4-b2b40e966312_isrForCrosstalkSources_1031720_60.stderr parsl.dataflow.dflow DEBUG: Task 2358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2358 has memoization hash 986a1dfd3ce6d3ebf85e6540fc1bc916 parsl.dataflow.memoization INFO: Task 2358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2358 try 0 launched on executor multi with executor id 2359 parsl.dataflow.dflow INFO: Standard output for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/3e8a2305-7e5c-493d-91b5-e0072e705492_isrForCrosstalkSources_1031730_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/3e8a2305-7e5c-493d-91b5-e0072e705492_isrForCrosstalkSources_1031730_4.stderr parsl.dataflow.dflow DEBUG: Task 2359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2359 has memoization hash fcf651b0eef22058c6d61c84a4046d79 parsl.dataflow.memoization INFO: Task 2359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5363e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2359 try 0 launched on executor multi with executor id 2360 parsl.dataflow.dflow INFO: Standard output for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/da824177-de8a-45e7-b763-0d7696c5af1f_isrForCrosstalkSources_1031724_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/da824177-de8a-45e7-b763-0d7696c5af1f_isrForCrosstalkSources_1031724_21.stderr parsl.dataflow.dflow DEBUG: Task 2360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2360 has memoization hash 2c83d6143c52503b470896a17f0b096c parsl.dataflow.memoization INFO: Task 2360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2360 try 0 launched on executor multi with executor id 2361 parsl.dataflow.dflow INFO: Standard output for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/755c73ec-9226-4a7f-aa83-eb34c2752a7b_isrForCrosstalkSources_1031699_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/755c73ec-9226-4a7f-aa83-eb34c2752a7b_isrForCrosstalkSources_1031699_6.stderr parsl.dataflow.dflow DEBUG: Task 2361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2361 has memoization hash 280d3550ade255762c60553977edc0de parsl.dataflow.memoization INFO: Task 2361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5365c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2361 try 0 launched on executor multi with executor id 2362 parsl.dataflow.dflow INFO: Standard output for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/58b316b2-5f37-4b9b-adb1-05e7cfe1c1a6_isrForCrosstalkSources_1031712_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/58b316b2-5f37-4b9b-adb1-05e7cfe1c1a6_isrForCrosstalkSources_1031712_2.stderr parsl.dataflow.dflow DEBUG: Task 2362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2362 has memoization hash 12b59b8b08e7d67619aa98ed257840e2 parsl.dataflow.memoization INFO: Task 2362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2362 try 0 launched on executor multi with executor id 2363 parsl.dataflow.dflow INFO: Standard output for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c251b4f8-308a-4c2b-a7fb-7a62bdd8fd0f_isrForCrosstalkSources_1031712_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c251b4f8-308a-4c2b-a7fb-7a62bdd8fd0f_isrForCrosstalkSources_1031712_19.stderr parsl.dataflow.dflow DEBUG: Task 2363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2363 has memoization hash 62ba7b846d0e90e519652193b511f209 parsl.dataflow.memoization INFO: Task 2363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2363 try 0 launched on executor multi with executor id 2364 parsl.dataflow.dflow INFO: Standard output for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/8ff8ccde-075a-4824-9e93-05f4bb3efd56_isrForCrosstalkSources_1031727_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/8ff8ccde-075a-4824-9e93-05f4bb3efd56_isrForCrosstalkSources_1031727_54.stderr parsl.dataflow.dflow DEBUG: Task 2364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2364 has memoization hash f9c38e7e8a6bcf946ec18620936b2f48 parsl.dataflow.memoization INFO: Task 2364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2364 try 0 launched on executor multi with executor id 2365 parsl.dataflow.dflow INFO: Standard output for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/1f9c347c-f72d-46c9-9f20-2e383895084c_isrForCrosstalkSources_1031727_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/1f9c347c-f72d-46c9-9f20-2e383895084c_isrForCrosstalkSources_1031727_7.stderr parsl.dataflow.dflow DEBUG: Task 2365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2365 has memoization hash eb71aea115d7e2f92733df0f925631a3 parsl.dataflow.memoization INFO: Task 2365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2365 try 0 launched on executor multi with executor id 2366 parsl.dataflow.dflow INFO: Standard output for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/39b31961-02c8-4e73-b47e-d4870450eaf6_isrForCrosstalkSources_1031738_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/39b31961-02c8-4e73-b47e-d4870450eaf6_isrForCrosstalkSources_1031738_33.stderr parsl.dataflow.dflow DEBUG: Task 2366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2366 has memoization hash 4ac36bb85174be62a16bda26c4f7a10d parsl.dataflow.memoization INFO: Task 2366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5368e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2366 try 0 launched on executor multi with executor id 2367 parsl.dataflow.dflow INFO: Standard output for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/90d7b8b1-3298-4c51-a0c1-35b9dbbbdbf4_isrForCrosstalkSources_1031729_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/90d7b8b1-3298-4c51-a0c1-35b9dbbbdbf4_isrForCrosstalkSources_1031729_11.stderr parsl.dataflow.dflow DEBUG: Task 2367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2367 has memoization hash 6cfba2f16cd7bd5da193f1ab71e58a36 parsl.dataflow.memoization INFO: Task 2367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5367a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2367 try 0 launched on executor multi with executor id 2368 parsl.dataflow.dflow INFO: Standard output for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/023e740b-8965-431c-8c9d-ca946cb145c6_isrForCrosstalkSources_1031734_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/023e740b-8965-431c-8c9d-ca946cb145c6_isrForCrosstalkSources_1031734_42.stderr parsl.dataflow.dflow DEBUG: Task 2368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2368 has memoization hash 91b6ce7f602cee2d2bdb268a749a039d parsl.dataflow.memoization INFO: Task 2368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2368 try 0 launched on executor multi with executor id 2369 parsl.dataflow.dflow INFO: Standard output for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/59ffdfa2-da80-4d5b-9447-d4088d5dd3b8_isrForCrosstalkSources_1031710_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/59ffdfa2-da80-4d5b-9447-d4088d5dd3b8_isrForCrosstalkSources_1031710_57.stderr parsl.dataflow.dflow DEBUG: Task 2369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2369 has memoization hash c9ed736307fdef21571143e18f178684 parsl.dataflow.memoization INFO: Task 2369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2369 try 0 launched on executor multi with executor id 2370 parsl.dataflow.dflow INFO: Standard output for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/38d0ef42-cd59-4d19-9944-9ee76e96b5d0_isrForCrosstalkSources_1031704_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/38d0ef42-cd59-4d19-9944-9ee76e96b5d0_isrForCrosstalkSources_1031704_27.stderr parsl.dataflow.dflow DEBUG: Task 2370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2370 has memoization hash a4e42ad57dfd392c884eed3586268be9 parsl.dataflow.memoization INFO: Task 2370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2370 try 0 launched on executor multi with executor id 2371 parsl.dataflow.dflow INFO: Standard output for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/06f34c6b-ad02-4e4b-9297-205ecba7cc51_isrForCrosstalkSources_1031702_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/06f34c6b-ad02-4e4b-9297-205ecba7cc51_isrForCrosstalkSources_1031702_58.stderr parsl.dataflow.dflow DEBUG: Task 2371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2371 has memoization hash 6c2ec5ef72bc61214c23b5e7d16cdce4 parsl.dataflow.memoization INFO: Task 2371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2371 try 0 launched on executor multi with executor id 2372 parsl.dataflow.dflow INFO: Standard output for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3f53872b-cd49-4495-a067-94a60b7f59f4_isrForCrosstalkSources_1031710_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3f53872b-cd49-4495-a067-94a60b7f59f4_isrForCrosstalkSources_1031710_49.stderr parsl.dataflow.dflow DEBUG: Task 2372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2372 has memoization hash 9ee9e50f0ae888c9d6f0fbd5990465c8 parsl.dataflow.memoization INFO: Task 2372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2372 try 0 launched on executor multi with executor id 2373 parsl.dataflow.dflow INFO: Standard output for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/eac392ab-b49d-472d-9749-2e47979a850b_isrForCrosstalkSources_1031717_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/eac392ab-b49d-472d-9749-2e47979a850b_isrForCrosstalkSources_1031717_53.stderr parsl.dataflow.dflow DEBUG: Task 2373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2373 has memoization hash e85861b560b9ca354e8822752687c4f4 parsl.dataflow.memoization INFO: Task 2373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2373 try 0 launched on executor multi with executor id 2374 parsl.dataflow.dflow INFO: Standard output for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b5beebf0-6f3f-4b5c-9984-2689d235a3a1_isrForCrosstalkSources_1031708_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b5beebf0-6f3f-4b5c-9984-2689d235a3a1_isrForCrosstalkSources_1031708_12.stderr parsl.dataflow.dflow DEBUG: Task 2374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2374 has memoization hash 173f30eaa1d964fb28a72c6e1be1e0bc parsl.dataflow.memoization INFO: Task 2374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2374 try 0 launched on executor multi with executor id 2375 parsl.dataflow.dflow INFO: Standard output for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/70af85b9-a2da-419e-a850-4521e8a071a3_isrForCrosstalkSources_1031724_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/70af85b9-a2da-419e-a850-4521e8a071a3_isrForCrosstalkSources_1031724_48.stderr parsl.dataflow.dflow DEBUG: Task 2375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2375 has memoization hash cf46c3884bfd1736cdead5daa3ff64c4 parsl.dataflow.memoization INFO: Task 2375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2375 try 0 launched on executor multi with executor id 2376 parsl.dataflow.dflow INFO: Standard output for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7a28086c-dca6-4861-ba1e-0d7f18d9db8a_isrForCrosstalkSources_1031725_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7a28086c-dca6-4861-ba1e-0d7f18d9db8a_isrForCrosstalkSources_1031725_46.stderr parsl.dataflow.dflow DEBUG: Task 2376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2376 has memoization hash 159407c3fea3a8072332076bb71e95da parsl.dataflow.memoization INFO: Task 2376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2376 try 0 launched on executor multi with executor id 2377 parsl.dataflow.dflow INFO: Standard output for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/b2c72f0c-ad7c-4170-bcf2-a47648abd309_isrForCrosstalkSources_1031709_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/b2c72f0c-ad7c-4170-bcf2-a47648abd309_isrForCrosstalkSources_1031709_30.stderr parsl.dataflow.dflow DEBUG: Task 2377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2377 has memoization hash d1a3ed06df706ac859f2668dd799e2f5 parsl.dataflow.memoization INFO: Task 2377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f536f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2377 try 0 launched on executor multi with executor id 2378 parsl.dataflow.dflow INFO: Standard output for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/eaa1549e-20ba-4a6c-ab08-4343e15e2b20_isrForCrosstalkSources_1031726_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/eaa1549e-20ba-4a6c-ab08-4343e15e2b20_isrForCrosstalkSources_1031726_34.stderr parsl.dataflow.dflow DEBUG: Task 2378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2378 has memoization hash 2b1a83ac2033911b5165f3637ff2ed62 parsl.dataflow.memoization INFO: Task 2378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2378 try 0 launched on executor multi with executor id 2379 parsl.dataflow.dflow INFO: Standard output for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/658810b7-189f-45d5-b6d3-46d560d7f003_isrForCrosstalkSources_1031719_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/658810b7-189f-45d5-b6d3-46d560d7f003_isrForCrosstalkSources_1031719_31.stderr parsl.dataflow.dflow DEBUG: Task 2379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2379 has memoization hash 5c426d9dcf4b5e00d0696fdb278bb1f1 parsl.dataflow.memoization INFO: Task 2379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2379 try 0 launched on executor multi with executor id 2380 parsl.dataflow.dflow INFO: Standard output for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/b22631c9-df58-47da-9b4d-bd6fa3ef7fcc_isrForCrosstalkSources_1031699_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/b22631c9-df58-47da-9b4d-bd6fa3ef7fcc_isrForCrosstalkSources_1031699_49.stderr parsl.dataflow.dflow DEBUG: Task 2380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2380 has memoization hash 9a56ece34d3cc5e2c80796bb2f7c163b parsl.dataflow.memoization INFO: Task 2380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2380 try 0 launched on executor multi with executor id 2381 parsl.dataflow.dflow INFO: Standard output for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/71c53733-a64e-4b2f-a88f-f7c4362f0538_isrForCrosstalkSources_1031703_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/71c53733-a64e-4b2f-a88f-f7c4362f0538_isrForCrosstalkSources_1031703_27.stderr parsl.dataflow.dflow DEBUG: Task 2381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2381 has memoization hash db0c505606f312ce668d7e4087c354f8 parsl.dataflow.memoization INFO: Task 2381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2381 try 0 launched on executor multi with executor id 2382 parsl.dataflow.dflow INFO: Standard output for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/9b62fbd3-e390-4ddf-a0f0-6ba6cd4a6fb0_isrForCrosstalkSources_1031713_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/9b62fbd3-e390-4ddf-a0f0-6ba6cd4a6fb0_isrForCrosstalkSources_1031713_19.stderr parsl.dataflow.dflow DEBUG: Task 2382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2382 has memoization hash 6943aba3f5506c72da9e4979ddafc587 parsl.dataflow.memoization INFO: Task 2382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2382 try 0 launched on executor multi with executor id 2383 parsl.dataflow.dflow INFO: Standard output for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/128a6f43-37fb-47c2-a8d2-394e16f100bd_isrForCrosstalkSources_1031719_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/128a6f43-37fb-47c2-a8d2-394e16f100bd_isrForCrosstalkSources_1031719_7.stderr parsl.dataflow.dflow DEBUG: Task 2383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2383 has memoization hash c069ea210291cfc23858becb1f9c99c7 parsl.dataflow.memoization INFO: Task 2383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2383 try 0 launched on executor multi with executor id 2384 parsl.dataflow.dflow INFO: Standard output for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/1039694b-25da-4b98-a066-c26552fa420e_isrForCrosstalkSources_1031723_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/1039694b-25da-4b98-a066-c26552fa420e_isrForCrosstalkSources_1031723_15.stderr parsl.dataflow.dflow DEBUG: Task 2384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2384 has memoization hash 11e81f94f722ed794fdf1d20256000d4 parsl.dataflow.memoization INFO: Task 2384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5379c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2384 try 0 launched on executor multi with executor id 2385 parsl.dataflow.dflow INFO: Standard output for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e346a0d7-3aa5-4fa2-aa84-12c1998f418c_isrForCrosstalkSources_1031717_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e346a0d7-3aa5-4fa2-aa84-12c1998f418c_isrForCrosstalkSources_1031717_20.stderr parsl.dataflow.dflow DEBUG: Task 2385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2385 has memoization hash 36358223100b62d919bc1bf35ecd71b4 parsl.dataflow.memoization INFO: Task 2385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2385 try 0 launched on executor multi with executor id 2386 parsl.dataflow.dflow INFO: Standard output for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c3294c85-cbb7-4a79-80e9-4e4036ac2ae1_isrForCrosstalkSources_1031718_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c3294c85-cbb7-4a79-80e9-4e4036ac2ae1_isrForCrosstalkSources_1031718_30.stderr parsl.dataflow.dflow DEBUG: Task 2386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2386 has memoization hash 96535e9461fd655fc7df13f4b57a784c parsl.dataflow.memoization INFO: Task 2386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5377e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2386 try 0 launched on executor multi with executor id 2387 parsl.dataflow.dflow INFO: Standard output for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/94a656dc-cc60-4b17-85b5-795369ac05fc_isrForCrosstalkSources_1031720_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/94a656dc-cc60-4b17-85b5-795369ac05fc_isrForCrosstalkSources_1031720_22.stderr parsl.dataflow.dflow DEBUG: Task 2387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2387 has memoization hash 79632fc7faa59d2cd89f14304d73e751 parsl.dataflow.memoization INFO: Task 2387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2387 try 0 launched on executor multi with executor id 2388 parsl.dataflow.dflow INFO: Standard output for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b8cba738-2e19-4d30-a596-d4b09936f52d_isrForCrosstalkSources_1031716_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b8cba738-2e19-4d30-a596-d4b09936f52d_isrForCrosstalkSources_1031716_4.stderr parsl.dataflow.dflow DEBUG: Task 2388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2388 has memoization hash 0d3062349494b1e7b550744cc9ec8106 parsl.dataflow.memoization INFO: Task 2388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2388 try 0 launched on executor multi with executor id 2389 parsl.dataflow.dflow INFO: Standard output for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c295276d-789c-4916-ad27-f15dfb2bf5fd_isrForCrosstalkSources_1031728_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c295276d-789c-4916-ad27-f15dfb2bf5fd_isrForCrosstalkSources_1031728_4.stderr parsl.dataflow.dflow DEBUG: Task 2389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2389 has memoization hash fe5c14beeef8a0c19c42f89aa826194a parsl.dataflow.memoization INFO: Task 2389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5376a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2389 try 0 launched on executor multi with executor id 2390 parsl.dataflow.dflow INFO: Standard output for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/fcf80a3c-727a-473b-b84e-25b78b95a4ca_isrForCrosstalkSources_1031718_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/fcf80a3c-727a-473b-b84e-25b78b95a4ca_isrForCrosstalkSources_1031718_29.stderr parsl.dataflow.dflow DEBUG: Task 2390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2390 has memoization hash 7196bf292ebee146ee19c6e7a028c2bf parsl.dataflow.memoization INFO: Task 2390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2390 try 0 launched on executor multi with executor id 2391 parsl.dataflow.dflow INFO: Standard output for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/55df23ed-0eec-4f4b-8a8a-4ab8d0dc047b_isrForCrosstalkSources_1031702_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/55df23ed-0eec-4f4b-8a8a-4ab8d0dc047b_isrForCrosstalkSources_1031702_23.stderr parsl.dataflow.dflow DEBUG: Task 2391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2391 has memoization hash 840bffcef71a2c7acd00384237df8bd2 parsl.dataflow.memoization INFO: Task 2391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5374c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2391 try 0 launched on executor multi with executor id 2392 parsl.dataflow.dflow INFO: Standard output for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/b10285b6-43fc-4a33-8ae2-75d1790f6d26_isrForCrosstalkSources_1031701_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/b10285b6-43fc-4a33-8ae2-75d1790f6d26_isrForCrosstalkSources_1031701_7.stderr parsl.dataflow.dflow DEBUG: Task 2392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2392 has memoization hash 5415a153630fa6ad2d31c4525106899f parsl.dataflow.memoization INFO: Task 2392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2392 try 0 launched on executor multi with executor id 2393 parsl.dataflow.dflow INFO: Standard output for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c57fd8ee-123a-47be-87c9-b12c936a20cb_isrForCrosstalkSources_1031734_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c57fd8ee-123a-47be-87c9-b12c936a20cb_isrForCrosstalkSources_1031734_15.stderr parsl.dataflow.dflow DEBUG: Task 2393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2393 has memoization hash b5a750eae6c826b1d375fdbb3f25d37f parsl.dataflow.memoization INFO: Task 2393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5372e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2393 try 0 launched on executor multi with executor id 2394 parsl.dataflow.dflow INFO: Standard output for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/89daeb8d-83dc-4a46-8f64-1fb68609cc9b_isrForCrosstalkSources_1031738_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/89daeb8d-83dc-4a46-8f64-1fb68609cc9b_isrForCrosstalkSources_1031738_43.stderr parsl.dataflow.dflow DEBUG: Task 2394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2394 has memoization hash b94370fde0fbad47af639594a1ea9ef4 parsl.dataflow.memoization INFO: Task 2394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2394 try 0 launched on executor multi with executor id 2395 parsl.dataflow.dflow INFO: Standard output for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/db5b87d0-45a6-475d-b8f2-d922b4fb03d2_isrForCrosstalkSources_1031699_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/db5b87d0-45a6-475d-b8f2-d922b4fb03d2_isrForCrosstalkSources_1031699_22.stderr parsl.dataflow.dflow DEBUG: Task 2395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2395 has memoization hash ec80f6bd69080810d9939b693f1ab6b5 parsl.dataflow.memoization INFO: Task 2395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2395 try 0 launched on executor multi with executor id 2396 parsl.dataflow.dflow INFO: Standard output for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/0119114b-7ae1-414a-bb99-0c1f6ddfd4a0_isrForCrosstalkSources_1031735_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/0119114b-7ae1-414a-bb99-0c1f6ddfd4a0_isrForCrosstalkSources_1031735_18.stderr parsl.dataflow.dflow DEBUG: Task 2396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2396 has memoization hash 2c995d700c91d970f9a9ee8da0931506 parsl.dataflow.memoization INFO: Task 2396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5371a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2396 try 0 launched on executor multi with executor id 2397 parsl.dataflow.dflow INFO: Standard output for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d3ebfbba-e37d-4d15-a46d-ed635cbd5616_isrForCrosstalkSources_1031722_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d3ebfbba-e37d-4d15-a46d-ed635cbd5616_isrForCrosstalkSources_1031722_29.stderr parsl.dataflow.dflow DEBUG: Task 2397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2397 has memoization hash d0739b62ab056bb4b6df964f5de19e42 parsl.dataflow.memoization INFO: Task 2397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f537100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2397 try 0 launched on executor multi with executor id 2398 parsl.dataflow.dflow INFO: Standard output for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b145da29-11b0-4cee-8035-16ad43371759_isrForCrosstalkSources_1031703_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b145da29-11b0-4cee-8035-16ad43371759_isrForCrosstalkSources_1031703_11.stderr parsl.dataflow.dflow DEBUG: Task 2398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2398 has memoization hash ea64835b4f43d2c6018da152db111a25 parsl.dataflow.memoization INFO: Task 2398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f534fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2398 try 0 launched on executor multi with executor id 2399 parsl.dataflow.dflow INFO: Standard output for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/08e101bc-16c2-41e7-89a8-5c831a27cfbc_isrForCrosstalkSources_1031712_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/08e101bc-16c2-41e7-89a8-5c831a27cfbc_isrForCrosstalkSources_1031712_36.stderr parsl.dataflow.dflow DEBUG: Task 2399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2399 has memoization hash 0c914ea070d13029747a01bd673e5dab parsl.dataflow.memoization INFO: Task 2399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5351c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2399 try 0 launched on executor multi with executor id 2400 parsl.dataflow.dflow INFO: Standard output for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b7fd96c3-debc-4c52-9ab4-b13e38ef7c39_isrForCrosstalkSources_1031714_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b7fd96c3-debc-4c52-9ab4-b13e38ef7c39_isrForCrosstalkSources_1031714_36.stderr parsl.dataflow.dflow DEBUG: Task 2400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2400 has memoization hash b076e01f08d27470e7607ab34139e6b9 parsl.dataflow.memoization INFO: Task 2400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2400 try 0 launched on executor multi with executor id 2401 parsl.dataflow.dflow INFO: Standard output for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/6c3ae3c1-f69f-4603-a20b-efe4d551afc6_isrForCrosstalkSources_1031734_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/6c3ae3c1-f69f-4603-a20b-efe4d551afc6_isrForCrosstalkSources_1031734_10.stderr parsl.dataflow.dflow DEBUG: Task 2401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2401 has memoization hash 9284dd74b3c0795c34cd16a5e0c26dbd parsl.dataflow.memoization INFO: Task 2401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2401 try 0 launched on executor multi with executor id 2402 parsl.dataflow.dflow INFO: Standard output for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/22cb06e0-9500-4299-b35e-6512bd46ea29_isrForCrosstalkSources_1031722_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/22cb06e0-9500-4299-b35e-6512bd46ea29_isrForCrosstalkSources_1031722_49.stderr parsl.dataflow.dflow DEBUG: Task 2402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2402 has memoization hash 3aaaccc65a7368cbb6ef33dc5e4ba0fb parsl.dataflow.memoization INFO: Task 2402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2402 try 0 launched on executor multi with executor id 2403 parsl.dataflow.dflow INFO: Standard output for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d281c170-51ea-4707-9453-e4e4d95e268f_isrForCrosstalkSources_1031730_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d281c170-51ea-4707-9453-e4e4d95e268f_isrForCrosstalkSources_1031730_40.stderr parsl.dataflow.dflow DEBUG: Task 2403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2403 has memoization hash 4c0c25af806b898449dbab7ed2a35823 parsl.dataflow.memoization INFO: Task 2403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2403 try 0 launched on executor multi with executor id 2404 parsl.dataflow.dflow INFO: Standard output for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4c8e195b-95eb-4aa5-8d38-1c1cc67c29c3_isrForCrosstalkSources_1031706_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4c8e195b-95eb-4aa5-8d38-1c1cc67c29c3_isrForCrosstalkSources_1031706_53.stderr parsl.dataflow.dflow DEBUG: Task 2404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2404 has memoization hash 4b4548379c396bbec58e18205bb4f171 parsl.dataflow.memoization INFO: Task 2404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2404 try 0 launched on executor multi with executor id 2405 parsl.dataflow.dflow INFO: Standard output for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1dd17c85-50dc-4e77-bf95-9f4d94d24b3f_isrForCrosstalkSources_1031709_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1dd17c85-50dc-4e77-bf95-9f4d94d24b3f_isrForCrosstalkSources_1031709_3.stderr parsl.dataflow.dflow DEBUG: Task 2405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2405 has memoization hash 9fd463fbe4e9a70a2ba134e8659caebc parsl.dataflow.memoization INFO: Task 2405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2405 try 0 launched on executor multi with executor id 2406 parsl.dataflow.dflow INFO: Standard output for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/4525466e-a8c4-4c26-8158-dc0524319dc0_isrForCrosstalkSources_1031702_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/4525466e-a8c4-4c26-8158-dc0524319dc0_isrForCrosstalkSources_1031702_29.stderr parsl.dataflow.dflow DEBUG: Task 2406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2406 has memoization hash 09ef5eec4a9968225e59ff0293d06cf2 parsl.dataflow.memoization INFO: Task 2406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2406 try 0 launched on executor multi with executor id 2407 parsl.dataflow.dflow INFO: Standard output for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c8239655-0eb0-45ef-bd0a-51a4c9f00f5a_isrForCrosstalkSources_1031734_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c8239655-0eb0-45ef-bd0a-51a4c9f00f5a_isrForCrosstalkSources_1031734_9.stderr parsl.dataflow.dflow DEBUG: Task 2407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2407 has memoization hash 2db64e412d3ee6ad6e8f73fbf0eed111 parsl.dataflow.memoization INFO: Task 2407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2407 try 0 launched on executor multi with executor id 2408 parsl.dataflow.dflow INFO: Standard output for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/d0ce25cf-b0c3-4bb2-a0a8-d0eea6d90144_isrForCrosstalkSources_1031724_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/d0ce25cf-b0c3-4bb2-a0a8-d0eea6d90144_isrForCrosstalkSources_1031724_7.stderr parsl.dataflow.dflow DEBUG: Task 2408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2408 has memoization hash 4a0e5da993a97f66f7900b5cd8cede2a parsl.dataflow.memoization INFO: Task 2408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2408 try 0 launched on executor multi with executor id 2409 parsl.dataflow.dflow INFO: Standard output for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/70709163-cb79-4c21-b161-5c61ba7d1de4_isrForCrosstalkSources_1031722_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/70709163-cb79-4c21-b161-5c61ba7d1de4_isrForCrosstalkSources_1031722_50.stderr parsl.dataflow.dflow DEBUG: Task 2409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2409 has memoization hash d81e1644b7345ce5a430fff37029c06a parsl.dataflow.memoization INFO: Task 2409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2409 try 0 launched on executor multi with executor id 2410 parsl.dataflow.dflow INFO: Standard output for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/9efcf06b-ed2f-4c1b-9e95-bdfafe729e0c_isrForCrosstalkSources_1031700_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/9efcf06b-ed2f-4c1b-9e95-bdfafe729e0c_isrForCrosstalkSources_1031700_16.stderr parsl.dataflow.dflow DEBUG: Task 2410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2410 has memoization hash 0a236b2d1ee3c01f75bc0beaa73efedc parsl.dataflow.memoization INFO: Task 2410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2410 try 0 launched on executor multi with executor id 2411 parsl.dataflow.dflow INFO: Standard output for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/69041da3-fe9f-42d1-b5f0-fff958c02876_isrForCrosstalkSources_1031735_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/69041da3-fe9f-42d1-b5f0-fff958c02876_isrForCrosstalkSources_1031735_60.stderr parsl.dataflow.dflow DEBUG: Task 2411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2411 has memoization hash c5a6f92e0360e329b0583ffa2b91fcc3 parsl.dataflow.memoization INFO: Task 2411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2411 try 0 launched on executor multi with executor id 2412 parsl.dataflow.dflow INFO: Standard output for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/1b05e552-6409-41e5-a383-3acc4704afeb_isrForCrosstalkSources_1031726_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/1b05e552-6409-41e5-a383-3acc4704afeb_isrForCrosstalkSources_1031726_56.stderr parsl.dataflow.dflow DEBUG: Task 2412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2412 has memoization hash d5e5320be33f23a20868c4d758c7e986 parsl.dataflow.memoization INFO: Task 2412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2412 try 0 launched on executor multi with executor id 2413 parsl.dataflow.dflow INFO: Standard output for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0f7a6ccc-f11a-4ded-a84c-005a83f8cfb3_isrForCrosstalkSources_1031707_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0f7a6ccc-f11a-4ded-a84c-005a83f8cfb3_isrForCrosstalkSources_1031707_29.stderr parsl.dataflow.dflow DEBUG: Task 2413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2413 has memoization hash c383469f660702b492afb11a7c3cf000 parsl.dataflow.memoization INFO: Task 2413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2413 try 0 launched on executor multi with executor id 2414 parsl.dataflow.dflow INFO: Standard output for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/869f46ec-4c00-4887-b290-46394c7afad2_isrForCrosstalkSources_1031715_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/869f46ec-4c00-4887-b290-46394c7afad2_isrForCrosstalkSources_1031715_3.stderr parsl.dataflow.dflow DEBUG: Task 2414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2414 has memoization hash 73bcffdca59b5ed814fe40f71cbd1805 parsl.dataflow.memoization INFO: Task 2414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2414 try 0 launched on executor multi with executor id 2415 parsl.dataflow.dflow INFO: Standard output for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/ebd03b2a-bedf-48e9-88fb-988bd35b37ab_isrForCrosstalkSources_1031704_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/ebd03b2a-bedf-48e9-88fb-988bd35b37ab_isrForCrosstalkSources_1031704_62.stderr parsl.dataflow.dflow DEBUG: Task 2415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2415 has memoization hash 27470d1176d203ffc65b49c6cb981bbd parsl.dataflow.memoization INFO: Task 2415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2415 try 0 launched on executor multi with executor id 2416 parsl.dataflow.dflow INFO: Standard output for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/c11cf64b-9c64-44ce-878d-b5189d6f788e_isrForCrosstalkSources_1031737_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/c11cf64b-9c64-44ce-878d-b5189d6f788e_isrForCrosstalkSources_1031737_20.stderr parsl.dataflow.dflow DEBUG: Task 2416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2416 has memoization hash 920c996876e26d980cd0a3953c3376e0 parsl.dataflow.memoization INFO: Task 2416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ccae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2416 try 0 launched on executor multi with executor id 2417 parsl.dataflow.dflow INFO: Standard output for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/20f02faa-2f46-408b-9e20-82689bca5ad0_isrForCrosstalkSources_1031726_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/20f02faa-2f46-408b-9e20-82689bca5ad0_isrForCrosstalkSources_1031726_11.stderr parsl.dataflow.dflow DEBUG: Task 2417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2417 has memoization hash 9973b5955d415a11b784e831368f93bf parsl.dataflow.memoization INFO: Task 2417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ccc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2417 try 0 launched on executor multi with executor id 2418 parsl.dataflow.dflow INFO: Standard output for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/0774314b-8463-470d-a7ad-83657c8fc6c9_isrForCrosstalkSources_1031710_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/0774314b-8463-470d-a7ad-83657c8fc6c9_isrForCrosstalkSources_1031710_15.stderr parsl.dataflow.dflow DEBUG: Task 2418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2418 has memoization hash ac71fe0aba607e2d6af8c1bb72d8fe86 parsl.dataflow.memoization INFO: Task 2418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ccb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2418 try 0 launched on executor multi with executor id 2419 parsl.dataflow.dflow INFO: Standard output for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/51b30b75-be33-4c34-8882-bc2617579d81_isrForCrosstalkSources_1031731_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/51b30b75-be33-4c34-8882-bc2617579d81_isrForCrosstalkSources_1031731_37.stderr parsl.dataflow.dflow DEBUG: Task 2419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2419 has memoization hash faf90b15c5cc5a22a6939a519244bc14 parsl.dataflow.memoization INFO: Task 2419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2419 try 0 launched on executor multi with executor id 2420 parsl.dataflow.dflow INFO: Standard output for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/11f62f69-4899-4ccf-9a12-dfe22a4cf8b9_isrForCrosstalkSources_1031703_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/11f62f69-4899-4ccf-9a12-dfe22a4cf8b9_isrForCrosstalkSources_1031703_36.stderr parsl.dataflow.dflow DEBUG: Task 2420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2420 has memoization hash c3b8090e5da9bfef74549e2dcc4e7c77 parsl.dataflow.memoization INFO: Task 2420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ccd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2420 try 0 launched on executor multi with executor id 2421 parsl.dataflow.dflow INFO: Standard output for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/69e9c047-be15-4848-b139-83e160817cde_isrForCrosstalkSources_1031736_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/69e9c047-be15-4848-b139-83e160817cde_isrForCrosstalkSources_1031736_47.stderr parsl.dataflow.dflow DEBUG: Task 2421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2421 has memoization hash 702c8caeaf53bc369a016a38525cafac parsl.dataflow.memoization INFO: Task 2421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2421 try 0 launched on executor multi with executor id 2422 parsl.dataflow.dflow INFO: Standard output for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/7181138f-8ddf-4a45-a2e0-e7904a3d2cf7_isrForCrosstalkSources_1031724_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/7181138f-8ddf-4a45-a2e0-e7904a3d2cf7_isrForCrosstalkSources_1031724_32.stderr parsl.dataflow.dflow DEBUG: Task 2422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2422 has memoization hash e7ac0892ea938de74fea97a115ba305b parsl.dataflow.memoization INFO: Task 2422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ccf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2422 try 0 launched on executor multi with executor id 2423 parsl.dataflow.dflow INFO: Standard output for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/d03dca95-35d3-4aaa-b80c-0be85303a9b8_isrForCrosstalkSources_1031716_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/d03dca95-35d3-4aaa-b80c-0be85303a9b8_isrForCrosstalkSources_1031716_60.stderr parsl.dataflow.dflow DEBUG: Task 2423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2423 has memoization hash 3fe71c001542d60754b75b666442743f parsl.dataflow.memoization INFO: Task 2423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ccea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2423 try 0 launched on executor multi with executor id 2424 parsl.dataflow.dflow INFO: Standard output for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b130dde1-b0e5-48ab-9ff8-d9d41b9859b9_isrForCrosstalkSources_1031712_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b130dde1-b0e5-48ab-9ff8-d9d41b9859b9_isrForCrosstalkSources_1031712_62.stderr parsl.dataflow.dflow DEBUG: Task 2424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2424 has memoization hash 21f4d71eba91e7ff030ea88d0be1408e parsl.dataflow.memoization INFO: Task 2424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ccfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2424 try 0 launched on executor multi with executor id 2425 parsl.dataflow.dflow INFO: Standard output for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3ad1a687-68e9-423c-9874-b7d04c758d51_isrForCrosstalkSources_1031719_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3ad1a687-68e9-423c-9874-b7d04c758d51_isrForCrosstalkSources_1031719_27.stderr parsl.dataflow.dflow DEBUG: Task 2425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2425 has memoization hash f20f3c26f37485e68293c4d275384672 parsl.dataflow.memoization INFO: Task 2425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2425 try 0 launched on executor multi with executor id 2426 parsl.dataflow.dflow INFO: Standard output for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/cbabcf33-d27e-4707-9cb1-a34a2b3926da_isrForCrosstalkSources_1031719_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/cbabcf33-d27e-4707-9cb1-a34a2b3926da_isrForCrosstalkSources_1031719_10.stderr parsl.dataflow.dflow DEBUG: Task 2426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2426 has memoization hash 88d21f97ccba3b9a5e87292452fa0383 parsl.dataflow.memoization INFO: Task 2426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2426 try 0 launched on executor multi with executor id 2427 parsl.dataflow.dflow INFO: Standard output for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/87b92d26-6127-4a7d-8c96-edbdbaef4b45_isrForCrosstalkSources_1031718_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/87b92d26-6127-4a7d-8c96-edbdbaef4b45_isrForCrosstalkSources_1031718_7.stderr parsl.dataflow.dflow DEBUG: Task 2427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2427 has memoization hash 4bf5bdbed36c864808df21a8f4f1fb5e parsl.dataflow.memoization INFO: Task 2427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2427 try 0 launched on executor multi with executor id 2428 parsl.dataflow.dflow INFO: Standard output for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4ca20430-aa16-4a65-99c3-871c2da8cc1e_isrForCrosstalkSources_1031717_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4ca20430-aa16-4a65-99c3-871c2da8cc1e_isrForCrosstalkSources_1031717_40.stderr parsl.dataflow.dflow DEBUG: Task 2428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2428 has memoization hash a798c50bf0dfdd7ab86d58485d6dfeb1 parsl.dataflow.memoization INFO: Task 2428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2428 try 0 launched on executor multi with executor id 2429 parsl.dataflow.dflow INFO: Standard output for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/baa393a5-a35b-4a11-800b-e6af21968eab_isrForCrosstalkSources_1031731_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/baa393a5-a35b-4a11-800b-e6af21968eab_isrForCrosstalkSources_1031731_40.stderr parsl.dataflow.dflow DEBUG: Task 2429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2429 has memoization hash e18853b03c18c1622392602e1780ec6c parsl.dataflow.memoization INFO: Task 2429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2429 try 0 launched on executor multi with executor id 2430 parsl.dataflow.dflow INFO: Standard output for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/cd8f8d7f-1f3f-4e96-8ba9-add12d0f2a80_isrForCrosstalkSources_1031735_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/cd8f8d7f-1f3f-4e96-8ba9-add12d0f2a80_isrForCrosstalkSources_1031735_37.stderr parsl.dataflow.dflow DEBUG: Task 2430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2430 has memoization hash 966edba09646ee6c6bff128e4aa05639 parsl.dataflow.memoization INFO: Task 2430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2430 try 0 launched on executor multi with executor id 2431 parsl.dataflow.dflow INFO: Standard output for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/2e2dee64-b2c2-4371-aae1-ac953330a3d1_isrForCrosstalkSources_1031737_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/2e2dee64-b2c2-4371-aae1-ac953330a3d1_isrForCrosstalkSources_1031737_21.stderr parsl.dataflow.dflow DEBUG: Task 2431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2431 has memoization hash 507f74546536b6b4c783411f37ee9fad parsl.dataflow.memoization INFO: Task 2431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2431 try 0 launched on executor multi with executor id 2432 parsl.dataflow.dflow INFO: Standard output for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/b7e9bc1f-02f7-43f5-9f19-6a5699929f90_isrForCrosstalkSources_1031710_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/b7e9bc1f-02f7-43f5-9f19-6a5699929f90_isrForCrosstalkSources_1031710_39.stderr parsl.dataflow.dflow DEBUG: Task 2432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2432 has memoization hash 6cc451908175337f621872ca9f0ca2ad parsl.dataflow.memoization INFO: Task 2432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2432 try 0 launched on executor multi with executor id 2433 parsl.dataflow.dflow INFO: Standard output for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/1e8cbec1-78da-4f4d-8350-2f1a08484502_isrForCrosstalkSources_1031712_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/1e8cbec1-78da-4f4d-8350-2f1a08484502_isrForCrosstalkSources_1031712_12.stderr parsl.dataflow.dflow DEBUG: Task 2433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2433 has memoization hash 8821c6a122bd9d33dd9f7031d520ee71 parsl.dataflow.memoization INFO: Task 2433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2433 try 0 launched on executor multi with executor id 2434 parsl.dataflow.dflow INFO: Standard output for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f1c3a7b1-2c61-4b62-bc6f-b72423f68424_isrForCrosstalkSources_1031723_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f1c3a7b1-2c61-4b62-bc6f-b72423f68424_isrForCrosstalkSources_1031723_52.stderr parsl.dataflow.dflow DEBUG: Task 2434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2434 has memoization hash aeb21b346ef819f17865098eaadccfd9 parsl.dataflow.memoization INFO: Task 2434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2434 try 0 launched on executor multi with executor id 2435 parsl.dataflow.dflow INFO: Standard output for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/9b0395c5-edc7-42e0-b330-7e12fd71c84b_isrForCrosstalkSources_1031700_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/9b0395c5-edc7-42e0-b330-7e12fd71c84b_isrForCrosstalkSources_1031700_13.stderr parsl.dataflow.dflow DEBUG: Task 2435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2435 has memoization hash 48b76349b2ddb9bfaf5ed6b8543f3225 parsl.dataflow.memoization INFO: Task 2435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2435 try 0 launched on executor multi with executor id 2436 parsl.dataflow.dflow INFO: Standard output for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/c286d528-78f0-4098-883b-34dde75f9a88_isrForCrosstalkSources_1031730_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/c286d528-78f0-4098-883b-34dde75f9a88_isrForCrosstalkSources_1031730_27.stderr parsl.dataflow.dflow DEBUG: Task 2436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2436 has memoization hash f4b189a15a91dc2481c29d6765087e40 parsl.dataflow.memoization INFO: Task 2436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2436 try 0 launched on executor multi with executor id 2437 parsl.dataflow.dflow INFO: Standard output for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7ce6cefb-b6b2-4867-8000-74e7f8235c21_isrForCrosstalkSources_1031702_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7ce6cefb-b6b2-4867-8000-74e7f8235c21_isrForCrosstalkSources_1031702_47.stderr parsl.dataflow.dflow DEBUG: Task 2437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2437 has memoization hash 3063d2bbb69751f3cef2ef31df7a5d17 parsl.dataflow.memoization INFO: Task 2437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2437 try 0 launched on executor multi with executor id 2438 parsl.dataflow.dflow INFO: Standard output for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/271db4f0-e91b-4145-99a7-58e46017954a_isrForCrosstalkSources_1031717_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/271db4f0-e91b-4145-99a7-58e46017954a_isrForCrosstalkSources_1031717_26.stderr parsl.dataflow.dflow DEBUG: Task 2438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2438 has memoization hash 648698f541d3c1aefb00f726bf828ba0 parsl.dataflow.memoization INFO: Task 2438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2438 try 0 launched on executor multi with executor id 2439 parsl.dataflow.dflow INFO: Standard output for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/1f060d83-339a-4775-a160-ce248f7243d0_isrForCrosstalkSources_1031736_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/1f060d83-339a-4775-a160-ce248f7243d0_isrForCrosstalkSources_1031736_37.stderr parsl.dataflow.dflow DEBUG: Task 2439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2439 has memoization hash 0da4c4bed5666559ef0b4604e15ec52b parsl.dataflow.memoization INFO: Task 2439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2439 try 0 launched on executor multi with executor id 2440 parsl.dataflow.dflow INFO: Standard output for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/78e9e8c9-d276-4028-b14c-eaaf79dd3e23_isrForCrosstalkSources_1031717_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/78e9e8c9-d276-4028-b14c-eaaf79dd3e23_isrForCrosstalkSources_1031717_14.stderr parsl.dataflow.dflow DEBUG: Task 2440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2440 has memoization hash 80944c3d01cd0e517eff11b8dc3ec469 parsl.dataflow.memoization INFO: Task 2440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2440 try 0 launched on executor multi with executor id 2441 parsl.dataflow.dflow INFO: Standard output for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1e154b85-4c07-4995-a7e6-28898078b2d4_isrForCrosstalkSources_1031700_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1e154b85-4c07-4995-a7e6-28898078b2d4_isrForCrosstalkSources_1031700_30.stderr parsl.dataflow.dflow DEBUG: Task 2441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2441 has memoization hash 9bd53ce6682946992ff38e186f2efe8c parsl.dataflow.memoization INFO: Task 2441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2441 try 0 launched on executor multi with executor id 2442 parsl.dataflow.dflow INFO: Standard output for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1d7ce7a5-45a9-40f7-844c-011e2e8937c7_isrForCrosstalkSources_1031703_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1d7ce7a5-45a9-40f7-844c-011e2e8937c7_isrForCrosstalkSources_1031703_58.stderr parsl.dataflow.dflow DEBUG: Task 2442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2442 has memoization hash 900aa97ada01945d7a69a9c654c4ac2e parsl.dataflow.memoization INFO: Task 2442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2442 try 0 launched on executor multi with executor id 2443 parsl.dataflow.dflow INFO: Standard output for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e828b06a-b768-40cf-9ac0-994a4a6c58d5_isrForCrosstalkSources_1031738_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e828b06a-b768-40cf-9ac0-994a4a6c58d5_isrForCrosstalkSources_1031738_24.stderr parsl.dataflow.dflow DEBUG: Task 2443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2443 has memoization hash bcae39da7e5404844e4fa78e003aaeef parsl.dataflow.memoization INFO: Task 2443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2443 try 0 launched on executor multi with executor id 2444 parsl.dataflow.dflow INFO: Standard output for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2e86bc71-9778-497d-8b5b-59184ed05393_isrForCrosstalkSources_1031726_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2e86bc71-9778-497d-8b5b-59184ed05393_isrForCrosstalkSources_1031726_51.stderr parsl.dataflow.dflow DEBUG: Task 2444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2444 has memoization hash 5a1fafab115daf717e5752aaf7a1271c parsl.dataflow.memoization INFO: Task 2444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2444 try 0 launched on executor multi with executor id 2445 parsl.dataflow.dflow INFO: Standard output for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/93956d27-79d4-475e-81b9-210ee3e66474_isrForCrosstalkSources_1031729_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/93956d27-79d4-475e-81b9-210ee3e66474_isrForCrosstalkSources_1031729_12.stderr parsl.dataflow.dflow DEBUG: Task 2445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2445 has memoization hash 2b0b97100124728b6a561ff75cdb2ee2 parsl.dataflow.memoization INFO: Task 2445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2445 try 0 launched on executor multi with executor id 2446 parsl.dataflow.dflow INFO: Standard output for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/570fffa8-c1ca-4a07-baef-bbaf98a94f38_isrForCrosstalkSources_1031735_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/570fffa8-c1ca-4a07-baef-bbaf98a94f38_isrForCrosstalkSources_1031735_53.stderr parsl.dataflow.dflow DEBUG: Task 2446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2446 has memoization hash 945d43fc6b8d6675180638d338c94143 parsl.dataflow.memoization INFO: Task 2446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2446 try 0 launched on executor multi with executor id 2447 parsl.dataflow.dflow INFO: Standard output for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/50767650-43f2-4de2-a48c-3c423542b066_isrForCrosstalkSources_1031708_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/50767650-43f2-4de2-a48c-3c423542b066_isrForCrosstalkSources_1031708_54.stderr parsl.dataflow.dflow DEBUG: Task 2447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2447 has memoization hash 76867cdc921ff3c43996fba126c4628c parsl.dataflow.memoization INFO: Task 2447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2447 try 0 launched on executor multi with executor id 2448 parsl.dataflow.dflow INFO: Standard output for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/56fe5671-9ea8-4a6b-ba5d-d5e16b03ec1e_isrForCrosstalkSources_1031710_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/56fe5671-9ea8-4a6b-ba5d-d5e16b03ec1e_isrForCrosstalkSources_1031710_19.stderr parsl.dataflow.dflow DEBUG: Task 2448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2448 has memoization hash 58948f6475a908a8b7e1d500a56a6f7c parsl.dataflow.memoization INFO: Task 2448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2448 try 0 launched on executor multi with executor id 2449 parsl.dataflow.dflow INFO: Standard output for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e536eddc-1da3-42b8-8c59-0c1a31fb0996_isrForCrosstalkSources_1031720_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e536eddc-1da3-42b8-8c59-0c1a31fb0996_isrForCrosstalkSources_1031720_48.stderr parsl.dataflow.dflow DEBUG: Task 2449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2449 has memoization hash 6d481e69bd56659259ec05db76790d6c parsl.dataflow.memoization INFO: Task 2449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2449 try 0 launched on executor multi with executor id 2450 parsl.dataflow.dflow INFO: Standard output for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/5a9f9c25-d1fa-480b-ba31-eb9da287bf21_isrForCrosstalkSources_1031725_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/5a9f9c25-d1fa-480b-ba31-eb9da287bf21_isrForCrosstalkSources_1031725_48.stderr parsl.dataflow.dflow DEBUG: Task 2450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2450 has memoization hash dd18a500120b18cf3567bc18b553b0f3 parsl.dataflow.memoization INFO: Task 2450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2450 try 0 launched on executor multi with executor id 2451 parsl.dataflow.dflow INFO: Standard output for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/36a2f6f2-379b-4fb1-a80b-f7e2eb7ce1fd_isrForCrosstalkSources_1031699_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/36a2f6f2-379b-4fb1-a80b-f7e2eb7ce1fd_isrForCrosstalkSources_1031699_47.stderr parsl.dataflow.dflow DEBUG: Task 2451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2451 has memoization hash e10c991f0919e4e43c8812c1b06321b7 parsl.dataflow.memoization INFO: Task 2451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2451 try 0 launched on executor multi with executor id 2452 parsl.dataflow.dflow INFO: Standard output for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2d6e5eaf-d713-45b6-97a5-142a24a131d8_isrForCrosstalkSources_1031709_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2d6e5eaf-d713-45b6-97a5-142a24a131d8_isrForCrosstalkSources_1031709_51.stderr parsl.dataflow.dflow DEBUG: Task 2452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2452 has memoization hash 42dbe71ff10b2e8dca4c4ffef980430d parsl.dataflow.memoization INFO: Task 2452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2452 try 0 launched on executor multi with executor id 2453 parsl.dataflow.dflow INFO: Standard output for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/8416ce41-6499-4a31-a880-c9c5965c6deb_isrForCrosstalkSources_1031710_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/8416ce41-6499-4a31-a880-c9c5965c6deb_isrForCrosstalkSources_1031710_27.stderr parsl.dataflow.dflow DEBUG: Task 2453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2453 has memoization hash e87ad20df55d8758361771367045143c parsl.dataflow.memoization INFO: Task 2453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2453 try 0 launched on executor multi with executor id 2454 parsl.dataflow.dflow INFO: Standard output for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/16518ba9-1593-4029-8c66-cb6f5efd3e4e_isrForCrosstalkSources_1031699_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/16518ba9-1593-4029-8c66-cb6f5efd3e4e_isrForCrosstalkSources_1031699_55.stderr parsl.dataflow.dflow DEBUG: Task 2454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2454 has memoization hash 4a153eacfafd1c341adc049601e0f29d parsl.dataflow.memoization INFO: Task 2454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2454 try 0 launched on executor multi with executor id 2455 parsl.dataflow.dflow INFO: Standard output for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/4c19818d-9146-49b4-9237-808a9b14d1ee_isrForCrosstalkSources_1031724_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/4c19818d-9146-49b4-9237-808a9b14d1ee_isrForCrosstalkSources_1031724_15.stderr parsl.dataflow.dflow DEBUG: Task 2455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2455 has memoization hash 0fb4262b36e0384c514f83b0835761e0 parsl.dataflow.memoization INFO: Task 2455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2455 try 0 launched on executor multi with executor id 2456 parsl.dataflow.dflow INFO: Standard output for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/1bfa01c3-aa56-4956-bcd6-072544c494ef_isrForCrosstalkSources_1031738_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/1bfa01c3-aa56-4956-bcd6-072544c494ef_isrForCrosstalkSources_1031738_54.stderr parsl.dataflow.dflow DEBUG: Task 2456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2456 has memoization hash 136ab2aa5fd7f0e6c079ba797298e262 parsl.dataflow.memoization INFO: Task 2456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2456 try 0 launched on executor multi with executor id 2457 parsl.dataflow.dflow INFO: Standard output for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/3e61b0d0-01e2-48fb-b69a-7fd08b72f7be_isrForCrosstalkSources_1031708_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/3e61b0d0-01e2-48fb-b69a-7fd08b72f7be_isrForCrosstalkSources_1031708_26.stderr parsl.dataflow.dflow DEBUG: Task 2457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2457 has memoization hash 063d9807a1ef039b6fe2600111d2d50c parsl.dataflow.memoization INFO: Task 2457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2457 try 0 launched on executor multi with executor id 2458 parsl.dataflow.dflow INFO: Standard output for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0fcd7c13-d27f-41aa-af8a-2b3f0be1f254_isrForCrosstalkSources_1031707_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0fcd7c13-d27f-41aa-af8a-2b3f0be1f254_isrForCrosstalkSources_1031707_36.stderr parsl.dataflow.dflow DEBUG: Task 2458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2458 has memoization hash f43ba138437218f70e04911a44c6df4e parsl.dataflow.memoization INFO: Task 2458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2458 try 0 launched on executor multi with executor id 2459 parsl.dataflow.dflow INFO: Standard output for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/637572c0-3333-45d1-966f-aa56107eb107_isrForCrosstalkSources_1031732_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/637572c0-3333-45d1-966f-aa56107eb107_isrForCrosstalkSources_1031732_28.stderr parsl.dataflow.dflow DEBUG: Task 2459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2459 has memoization hash ae54bbbabb58e0e2fdf504db2d0e8072 parsl.dataflow.memoization INFO: Task 2459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2459 try 0 launched on executor multi with executor id 2460 parsl.dataflow.dflow INFO: Standard output for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b576cf42-f0b0-4525-a1b6-6ab879563651_isrForCrosstalkSources_1031708_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b576cf42-f0b0-4525-a1b6-6ab879563651_isrForCrosstalkSources_1031708_24.stderr parsl.dataflow.dflow DEBUG: Task 2460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2460 has memoization hash 6bb124ed8c7a5e454ab9fd693abb0ade parsl.dataflow.memoization INFO: Task 2460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2460 try 0 launched on executor multi with executor id 2461 parsl.dataflow.dflow INFO: Standard output for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d3d61066-d678-4e8b-857d-4f04813e2b1b_isrForCrosstalkSources_1031722_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d3d61066-d678-4e8b-857d-4f04813e2b1b_isrForCrosstalkSources_1031722_21.stderr parsl.dataflow.dflow DEBUG: Task 2461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2461 has memoization hash cd8d49a21720ce112c135e0a79be74ab parsl.dataflow.memoization INFO: Task 2461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2461 try 0 launched on executor multi with executor id 2462 parsl.dataflow.dflow INFO: Standard output for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/13d324b4-bfc5-44c9-a632-cf1d5d025963_isrForCrosstalkSources_1031715_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/13d324b4-bfc5-44c9-a632-cf1d5d025963_isrForCrosstalkSources_1031715_18.stderr parsl.dataflow.dflow DEBUG: Task 2462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2462 has memoization hash 758a35964aecc6ac2936c52aa59b13d0 parsl.dataflow.memoization INFO: Task 2462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2462 try 0 launched on executor multi with executor id 2463 parsl.dataflow.dflow INFO: Standard output for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/687725ec-ba68-4dc8-ac3b-dd67fdc4b163_isrForCrosstalkSources_1031700_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/687725ec-ba68-4dc8-ac3b-dd67fdc4b163_isrForCrosstalkSources_1031700_52.stderr parsl.dataflow.dflow DEBUG: Task 2463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2463 has memoization hash 4a8375a2d5bb3b5538c92f6ae8b644b5 parsl.dataflow.memoization INFO: Task 2463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2463 try 0 launched on executor multi with executor id 2464 parsl.dataflow.dflow INFO: Standard output for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/93257301-af67-4b6e-8202-fa5acd18042f_isrForCrosstalkSources_1031730_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/93257301-af67-4b6e-8202-fa5acd18042f_isrForCrosstalkSources_1031730_47.stderr parsl.dataflow.dflow DEBUG: Task 2464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2464 has memoization hash c0fc708d3d5d09c2c90c05761d302be9 parsl.dataflow.memoization INFO: Task 2464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2464 try 0 launched on executor multi with executor id 2465 parsl.dataflow.dflow INFO: Standard output for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3011c9a9-3b26-4394-820c-8e89e45b2318_isrForCrosstalkSources_1031710_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3011c9a9-3b26-4394-820c-8e89e45b2318_isrForCrosstalkSources_1031710_11.stderr parsl.dataflow.dflow DEBUG: Task 2465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2465 has memoization hash 9dd85c640e26b9c6f64c5bd0005dbf69 parsl.dataflow.memoization INFO: Task 2465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ce980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2465 try 0 launched on executor multi with executor id 2466 parsl.dataflow.dflow INFO: Standard output for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/866ccc3a-8dbb-4f87-82d8-4b8c2742436f_isrForCrosstalkSources_1031725_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/866ccc3a-8dbb-4f87-82d8-4b8c2742436f_isrForCrosstalkSources_1031725_45.stderr parsl.dataflow.dflow DEBUG: Task 2466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2466 has memoization hash 5770c0573d1c71831d09fed730b20f9f parsl.dataflow.memoization INFO: Task 2466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2466 try 0 launched on executor multi with executor id 2467 parsl.dataflow.dflow INFO: Standard output for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a5537f65-d1a5-4154-b8bb-5eb463d9d4fe_isrForCrosstalkSources_1031725_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a5537f65-d1a5-4154-b8bb-5eb463d9d4fe_isrForCrosstalkSources_1031725_53.stderr parsl.dataflow.dflow DEBUG: Task 2467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2467 has memoization hash b8e302192a98720756fcfe4b2e065c44 parsl.dataflow.memoization INFO: Task 2467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ceb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2467 try 0 launched on executor multi with executor id 2468 parsl.dataflow.dflow INFO: Standard output for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/3d8e5ee7-28cf-4481-879e-792b70323bb9_isrForCrosstalkSources_1031720_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/3d8e5ee7-28cf-4481-879e-792b70323bb9_isrForCrosstalkSources_1031720_3.stderr parsl.dataflow.dflow DEBUG: Task 2468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2468 has memoization hash 9e42196e6d089af6acef56267fc43919 parsl.dataflow.memoization INFO: Task 2468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ceac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2468 try 0 launched on executor multi with executor id 2469 parsl.dataflow.dflow INFO: Standard output for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/98915ee4-2029-4271-950f-5493ed2f6c28_isrForCrosstalkSources_1031708_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/98915ee4-2029-4271-950f-5493ed2f6c28_isrForCrosstalkSources_1031708_7.stderr parsl.dataflow.dflow DEBUG: Task 2469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2469 has memoization hash 28b6ae015b063b9feb42f585a545d067 parsl.dataflow.memoization INFO: Task 2469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2469 try 0 launched on executor multi with executor id 2470 parsl.dataflow.dflow INFO: Standard output for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/a67c1242-be4b-4880-85a3-49aa85a3dbaf_isrForCrosstalkSources_1031737_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/a67c1242-be4b-4880-85a3-49aa85a3dbaf_isrForCrosstalkSources_1031737_4.stderr parsl.dataflow.dflow DEBUG: Task 2470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2470 has memoization hash b7d177b496183c5396eabea0b159b6cd parsl.dataflow.memoization INFO: Task 2470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5ceca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2470 try 0 launched on executor multi with executor id 2471 parsl.dataflow.dflow INFO: Standard output for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/16acf4fd-65d7-4626-bceb-7c3aaa476ea8_isrForCrosstalkSources_1031736_61.stdout 2024-03-17 04:02:26 proc_lsst.multi:170 [INFO] [multi] job_id 17147959 proc_lsst.multi INFO: [multi] job_id 17147959 2024-03-17 04:02:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-17 04:02:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147959 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147959 parsl.dataflow.dflow INFO: Standard error for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/16acf4fd-65d7-4626-bceb-7c3aaa476ea8_isrForCrosstalkSources_1031736_61.stderr parsl.dataflow.dflow DEBUG: Task 2471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2471 has memoization hash 4becee01fd2dedc7265339b192b28dec parsl.dataflow.memoization INFO: Task 2471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f70d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17147959 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-17 04:02:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-17 04:02:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-17 04:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2471 try 0 launched on executor multi with executor id 2472 parsl.dataflow.dflow INFO: Standard output for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d142063e-daf5-47d6-acec-dd125a8980fb_isrForCrosstalkSources_1031722_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d142063e-daf5-47d6-acec-dd125a8980fb_isrForCrosstalkSources_1031722_7.stderr parsl.dataflow.dflow DEBUG: Task 2472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2472 has memoization hash f558c607cf60e05a85aea23d21c47102 parsl.dataflow.memoization INFO: Task 2472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f3e4fb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2472 try 0 launched on executor multi with executor id 2473 parsl.dataflow.dflow INFO: Standard output for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/128cf33d-641a-4430-9c39-7186b5b53e5d_isrForCrosstalkSources_1031729_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/128cf33d-641a-4430-9c39-7186b5b53e5d_isrForCrosstalkSources_1031729_13.stderr parsl.dataflow.dflow DEBUG: Task 2473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2473 has memoization hash 5153c5b371853d1d5487915d9e2d57fc parsl.dataflow.memoization INFO: Task 2473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f44a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2473 try 0 launched on executor multi with executor id 2474 parsl.dataflow.dflow INFO: Standard output for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bc83decb-dd28-4301-abad-3dc5db9aa7ac_isrForCrosstalkSources_1031723_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bc83decb-dd28-4301-abad-3dc5db9aa7ac_isrForCrosstalkSources_1031723_60.stderr parsl.dataflow.dflow DEBUG: Task 2474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2474 has memoization hash c169df9a39be206dc1ba0b1d14dfa90f parsl.dataflow.memoization INFO: Task 2474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f449f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2474 try 0 launched on executor multi with executor id 2475 parsl.dataflow.dflow INFO: Standard output for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/737cf127-4525-4b27-af16-fe28713d8691_isrForCrosstalkSources_1031722_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/737cf127-4525-4b27-af16-fe28713d8691_isrForCrosstalkSources_1031722_35.stderr parsl.dataflow.dflow DEBUG: Task 2475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2475 has memoization hash d5bd9577379b37f8598c352b34c2d7d9 parsl.dataflow.memoization INFO: Task 2475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2475 try 0 launched on executor multi with executor id 2476 parsl.dataflow.dflow INFO: Standard output for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a9d0fd0e-557f-43c3-a9e7-bced034d41a5_isrForCrosstalkSources_1031709_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a9d0fd0e-557f-43c3-a9e7-bced034d41a5_isrForCrosstalkSources_1031709_22.stderr parsl.dataflow.dflow DEBUG: Task 2476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2476 has memoization hash 0681d679b795b1c33253e8e6fd479494 parsl.dataflow.memoization INFO: Task 2476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2476 try 0 launched on executor multi with executor id 2477 parsl.dataflow.dflow INFO: Standard output for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/406ced21-aeae-4e02-afda-9b86e00b8fc9_isrForCrosstalkSources_1031699_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/406ced21-aeae-4e02-afda-9b86e00b8fc9_isrForCrosstalkSources_1031699_23.stderr parsl.dataflow.dflow DEBUG: Task 2477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2477 has memoization hash 787c77d481b318f774c43e69c8471adf parsl.dataflow.memoization INFO: Task 2477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2477 try 0 launched on executor multi with executor id 2478 parsl.dataflow.dflow INFO: Standard output for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/05adaa7a-2c5e-42ac-aa08-04c10c42aa8a_isrForCrosstalkSources_1031736_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/05adaa7a-2c5e-42ac-aa08-04c10c42aa8a_isrForCrosstalkSources_1031736_27.stderr parsl.dataflow.dflow DEBUG: Task 2478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2478 has memoization hash 451de7f04cd39d9683897977bf3f6563 parsl.dataflow.memoization INFO: Task 2478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2478 try 0 launched on executor multi with executor id 2479 parsl.dataflow.dflow INFO: Standard output for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/27ecc693-e74f-4489-9704-4f6ca656bb9c_isrForCrosstalkSources_1031700_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/27ecc693-e74f-4489-9704-4f6ca656bb9c_isrForCrosstalkSources_1031700_12.stderr parsl.dataflow.dflow DEBUG: Task 2479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2479 has memoization hash 95d6d914895ffecdfa51977cc62e2a87 parsl.dataflow.memoization INFO: Task 2479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f2f5cf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2479 try 0 launched on executor multi with executor id 2480 parsl.dataflow.dflow INFO: Standard output for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/db6a62ef-597e-4eed-b96f-6c727b3a085a_isrForCrosstalkSources_1031701_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/db6a62ef-597e-4eed-b96f-6c727b3a085a_isrForCrosstalkSources_1031701_10.stderr 2024-03-17 04:02:27 proc_lsst.multi:170 [INFO] [multi] job_id 17147960 proc_lsst.multi INFO: [multi] job_id 17147960 2024-03-17 04:02:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-17 04:02:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147960 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147960 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17147960 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-17 04:02:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-17 04:02:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-17 04:02:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1b898efac2c4bc4d246f10d8f9dec67 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/328d0662-866f-4033-897b-b1a102bd9e87_isrForCrosstalkSources_1031712_4.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/328d0662-866f-4033-897b-b1a102bd9e87_isrForCrosstalkSources_1031712_4.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df98fab29598bfc65007b10fa3ed131c with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/ed3444fe-d064-427f-bfb6-8a0713744bf2_isrForCrosstalkSources_1031730_42.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/ed3444fe-d064-427f-bfb6-8a0713744bf2_isrForCrosstalkSources_1031730_42.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48501ef9f3e3ee956fe11edc4598c8b8 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ea54b745-c64d-4d9b-9deb-605005b8f31a_isrForCrosstalkSources_1031729_42.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ea54b745-c64d-4d9b-9deb-605005b8f31a_isrForCrosstalkSources_1031729_42.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61c3a88b8c412abd339f686c4ab13e33 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/73d22196-a6ed-4c86-b7ad-7c735c5a1be1_isrForCrosstalkSources_1031728_49.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/73d22196-a6ed-4c86-b7ad-7c735c5a1be1_isrForCrosstalkSources_1031728_49.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c39ad8a36a51b659d2e62f29fad69fb8 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fe4aa082-816f-45cc-8fd0-cac0fac0f196_isrForCrosstalkSources_1031723_57.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fe4aa082-816f-45cc-8fd0-cac0fac0f196_isrForCrosstalkSources_1031723_57.stderr 2024-03-17 04:02:30 proc_lsst.multi:170 [INFO] [multi] job_id 17147961 proc_lsst.multi INFO: [multi] job_id 17147961 2024-03-17 04:02:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:02:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:30 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17147961 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17147961 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17147961 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-17 04:02:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-17 04:02:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:02:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 04:02:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:02:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 335f76650980c7089a5dd03aaa17820f with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/69b6adab-0cf7-4c8b-a9a8-bfb1941f9bd0_isrForCrosstalkSources_1031733_40.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/69b6adab-0cf7-4c8b-a9a8-bfb1941f9bd0_isrForCrosstalkSources_1031733_40.stderr 2024-03-17 04:02:31 proc_lsst.multi:170 [INFO] [multi] job_id 17147962 proc_lsst.multi INFO: [multi] job_id 17147962 2024-03-17 04:02:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:02:31 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17147962 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17147962 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17147962 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-17 04:02:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-17 04:02:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:02:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:02:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bd49612de92748a2b1554b146f63613 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bbd5630d-f6d4-4a54-957b-6c3972d48d76_isrForCrosstalkSources_1031734_52.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bbd5630d-f6d4-4a54-957b-6c3972d48d76_isrForCrosstalkSources_1031734_52.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7fd24cbf854e3fcdb076b43e92fcfac with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/65e0eb97-13d4-438d-910b-f9b4699b03c4_isrForCrosstalkSources_1031711_29.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/65e0eb97-13d4-438d-910b-f9b4699b03c4_isrForCrosstalkSources_1031711_29.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6d25e0a4aa848653b37b5fad40cbff1 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4b170816-0ab1-4e89-9ab6-326a27076be5_isrForCrosstalkSources_1031723_40.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4b170816-0ab1-4e89-9ab6-326a27076be5_isrForCrosstalkSources_1031723_40.stderr 2024-03-17 04:02:34 proc_lsst.multi:170 [INFO] [multi] job_id 17147963 proc_lsst.multi INFO: [multi] job_id 17147963 2024-03-17 04:02:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 04:02:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:02:34 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17147963 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17147963 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17147963 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-17 04:02:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-17 04:02:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:02:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 04:02:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:02:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 319f830bffac7ab3d5b7b95b43ee01b2 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/389fe42a-eab8-4114-a589-de23385f57ab_isrForCrosstalkSources_1031738_25.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/389fe42a-eab8-4114-a589-de23385f57ab_isrForCrosstalkSources_1031738_25.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82b679cd46e90bc7be81ad0363c16aa4 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/97e0e125-8de8-4ffa-93f1-78b1d69e606b_isrForCrosstalkSources_1031718_39.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/97e0e125-8de8-4ffa-93f1-78b1d69e606b_isrForCrosstalkSources_1031718_39.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9590097fc0cf2a53d37a8518d6692f17 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/359e5de3-6aac-4d34-99f3-f507f6af16a0_isrForCrosstalkSources_1031706_46.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/359e5de3-6aac-4d34-99f3-f507f6af16a0_isrForCrosstalkSources_1031706_46.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1510d1040c7ee01221ffbd12e6063f79 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/08338d78-fe07-4f7c-acef-40fbc6490471_isrForCrosstalkSources_1031732_24.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/08338d78-fe07-4f7c-acef-40fbc6490471_isrForCrosstalkSources_1031732_24.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65080ce4acff022c7e2110ee47eb7ffc with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d85bfa30-01c0-4434-b9a3-7dc1df711e81_isrForCrosstalkSources_1031704_9.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d85bfa30-01c0-4434-b9a3-7dc1df711e81_isrForCrosstalkSources_1031704_9.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91131e2f7943013735898858aae03703 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/60161c13-a619-40f9-8e61-5a38f589f87c_isrForCrosstalkSources_1031730_57.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/60161c13-a619-40f9-8e61-5a38f589f87c_isrForCrosstalkSources_1031730_57.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00899f80e40d81be6b71b69ae197b671 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/2c34f7d0-68b9-4ed4-add6-078db2c929bf_isrForCrosstalkSources_1031733_26.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/2c34f7d0-68b9-4ed4-add6-078db2c929bf_isrForCrosstalkSources_1031733_26.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07d6ecf9800ea2a1168b37896aa1360d with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fac45196-306a-4d63-b9a6-8784b1cb1509_isrForCrosstalkSources_1031707_37.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fac45196-306a-4d63-b9a6-8784b1cb1509_isrForCrosstalkSources_1031707_37.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b393cf0cb92439fd5c17c98cff0f751a with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/c77cb30f-bff6-4d7b-8079-7dad33b2d45d_isrForCrosstalkSources_1031735_24.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/c77cb30f-bff6-4d7b-8079-7dad33b2d45d_isrForCrosstalkSources_1031735_24.stderr 2024-03-17 04:02:36 proc_lsst.multi:170 [INFO] [multi] job_id 17147964 proc_lsst.multi INFO: [multi] job_id 17147964 2024-03-17 04:02:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 04:02:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:02:36 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17147964 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17147964 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17147964 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-17 04:02:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-17 04:02:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:02:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 04:02:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:02:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 958f15b48deb95943a09039fbacb888b with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/186f677c-c6ab-4710-bd33-2603b21047ba_isrForCrosstalkSources_1031722_56.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/186f677c-c6ab-4710-bd33-2603b21047ba_isrForCrosstalkSources_1031722_56.stderr 2024-03-17 04:02:37 proc_lsst.multi:170 [INFO] [multi] job_id 17147965 proc_lsst.multi INFO: [multi] job_id 17147965 2024-03-17 04:02:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 04:02:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:02:37 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17147965 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17147965 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17147965 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-17 04:02:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-17 04:02:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:02:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:02:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 04:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:02:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:02:39 proc_lsst.multi:170 [INFO] [multi] job_id 17147966 proc_lsst.multi INFO: [multi] job_id 17147966 2024-03-17 04:02:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:02:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:02:39 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17147966 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17147966 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17147966 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22674350809360 2024-03-17 04:02:39 proc_lsst.multi:146 [INFO] found job 3544 in provider local proc_lsst.multi INFO: found job 3544 in provider local 2024-03-17 04:02:39 proc_lsst.multi:146 [INFO] found job 17147931 in provider astro proc_lsst.multi INFO: found job 17147931 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d6ad5f53a733117ece208d2504f128c with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/5123f9c5-cdf8-4bdf-874b-91ed685ba162_isrForCrosstalkSources_1031702_44.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/5123f9c5-cdf8-4bdf-874b-91ed685ba162_isrForCrosstalkSources_1031702_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:02:40 proc_lsst.multi:146 [INFO] found job 17147932 in provider astro proc_lsst.multi INFO: found job 17147932 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84ef2834825c5f171d10019fec0709d0 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/101f16e4-64f1-4945-9517-f4df847c3648_isrForCrosstalkSources_1031707_8.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/101f16e4-64f1-4945-9517-f4df847c3648_isrForCrosstalkSources_1031707_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:02:42 proc_lsst.multi:146 [INFO] found job 17147933 in provider astro proc_lsst.multi INFO: found job 17147933 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fb45831ed9010834caa434cd3c33d7c with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/41c7cb66-83e4-492a-8fe5-b70a9a9d9721_isrForCrosstalkSources_1031708_23.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/41c7cb66-83e4-492a-8fe5-b70a9a9d9721_isrForCrosstalkSources_1031708_23.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bdf1f680c7306cd616e5808a18766cc with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/c5466470-615e-4c44-8973-4fc8dd781e2f_isrForCrosstalkSources_1031724_5.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/c5466470-615e-4c44-8973-4fc8dd781e2f_isrForCrosstalkSources_1031724_5.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18ef61918fad723274629bfb132314b6 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/df5ebb1d-5738-4db4-a38b-ed869f6b52a3_isrForCrosstalkSources_1031724_44.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/df5ebb1d-5738-4db4-a38b-ed869f6b52a3_isrForCrosstalkSources_1031724_44.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16053197672ea7601fcf5a493655bdc6 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8be06254-36b6-4377-b6d3-5ee1095a64b1_isrForCrosstalkSources_1031732_34.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8be06254-36b6-4377-b6d3-5ee1095a64b1_isrForCrosstalkSources_1031732_34.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc0c15a6783683b5e08d83c0fb0e7d5a with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d275a19c-0176-4ddf-9692-15fdbbb37145_isrForCrosstalkSources_1031722_3.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d275a19c-0176-4ddf-9692-15fdbbb37145_isrForCrosstalkSources_1031722_3.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c46566f38d5f9f86545e2132efa28e9e with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0144e49d-2658-4678-9af1-c15a777d2cb5_isrForCrosstalkSources_1031706_36.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0144e49d-2658-4678-9af1-c15a777d2cb5_isrForCrosstalkSources_1031706_36.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f2ced48cd8f8925e6f1da9e8569ece6 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0433a00a-e723-400e-9803-ab2068206017_isrForCrosstalkSources_1031714_22.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0433a00a-e723-400e-9803-ab2068206017_isrForCrosstalkSources_1031714_22.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c33875519b51dfe76f6cdcb927c83cd with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6d67fb17-77e0-4536-957a-d4d7df5d96c2_isrForCrosstalkSources_1031737_15.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6d67fb17-77e0-4536-957a-d4d7df5d96c2_isrForCrosstalkSources_1031737_15.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 854043bd57cbc937b167fa429ef11bd8 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/7a2c9bc3-08d6-41e0-b23b-14483e0ad157_isrForCrosstalkSources_1031715_1.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/7a2c9bc3-08d6-41e0-b23b-14483e0ad157_isrForCrosstalkSources_1031715_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:02:44 proc_lsst.multi:146 [INFO] found job 17147934 in provider astro proc_lsst.multi INFO: found job 17147934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5259ae2040126e0e9acaf4dd17115d6f with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b3e9beff-b14c-4e45-8b31-fd6bb4e52e3c_isrForCrosstalkSources_1031721_14.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b3e9beff-b14c-4e45-8b31-fd6bb4e52e3c_isrForCrosstalkSources_1031721_14.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1177f22a490d6d73aa3230b7b5e50067 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d85985e2-8754-4c2e-b3e3-c5354b9fdef3_isrForCrosstalkSources_1031714_21.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d85985e2-8754-4c2e-b3e3-c5354b9fdef3_isrForCrosstalkSources_1031714_21.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65ba70137ba20ccb1d8d5f895d137d04 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/34ca5f48-d1fe-4ec7-93e8-42740c4e0a4b_isrForCrosstalkSources_1031717_21.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/34ca5f48-d1fe-4ec7-93e8-42740c4e0a4b_isrForCrosstalkSources_1031717_21.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57dd832035ea240010a5d2667c4c65ec with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2b1db993-5303-4771-8721-e652d95b8514_isrForCrosstalkSources_1031703_21.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2b1db993-5303-4771-8721-e652d95b8514_isrForCrosstalkSources_1031703_21.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdf60f7ae8f9317bebca5a641baac1d8 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4ed70c82-25bf-4d29-be9b-8f089925774d_isrForCrosstalkSources_1031723_56.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4ed70c82-25bf-4d29-be9b-8f089925774d_isrForCrosstalkSources_1031723_56.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e160de82112de6126ca54944e6737768 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/08a4c632-a4bd-43e9-9b07-c36d382cc145_isrForCrosstalkSources_1031728_28.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/08a4c632-a4bd-43e9-9b07-c36d382cc145_isrForCrosstalkSources_1031728_28.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b13212c31d0b9638af4576b86d41c610 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4fd6d74d-34ea-42be-bb9d-b48781448a96_isrForCrosstalkSources_1031703_7.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4fd6d74d-34ea-42be-bb9d-b48781448a96_isrForCrosstalkSources_1031703_7.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a917998773435c5a7f09f8a0a140fc2 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/e76be664-4ce3-41e7-a703-1ed4925ad120_isrForCrosstalkSources_1031726_37.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/e76be664-4ce3-41e7-a703-1ed4925ad120_isrForCrosstalkSources_1031726_37.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d49cc28f8ae01c29fb0f7878d20064d with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a0aaa727-ac70-4b60-9708-efe18aa9c584_isrForCrosstalkSources_1031725_11.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a0aaa727-ac70-4b60-9708-efe18aa9c584_isrForCrosstalkSources_1031725_11.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37e44537bf3e7bad5331023e47145e2b with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/57190881-7df4-40fa-9e5e-ab5881d058cb_isrForCrosstalkSources_1031704_24.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/57190881-7df4-40fa-9e5e-ab5881d058cb_isrForCrosstalkSources_1031704_24.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aef1a1d791a31f0dd8e7d1142e015d14 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/cde51801-d464-40f1-84d3-38eaa02d5900_isrForCrosstalkSources_1031704_16.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/cde51801-d464-40f1-84d3-38eaa02d5900_isrForCrosstalkSources_1031704_16.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ae7c71ce9bc88ee31c5b094f0c6e4de with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/dde40872-abf3-4aff-b47b-758776c4087f_isrForCrosstalkSources_1031706_33.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/dde40872-abf3-4aff-b47b-758776c4087f_isrForCrosstalkSources_1031706_33.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 243c36322a30966b08f05dfb07c8e96d with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/793f3a52-291f-4b2e-9760-391472ce2734_isrForCrosstalkSources_1031713_9.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/793f3a52-291f-4b2e-9760-391472ce2734_isrForCrosstalkSources_1031713_9.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 683388fb4901c39640477ae7b78c176c with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/52c4a6b0-5455-46b0-8105-c2377b4aab1f_isrForCrosstalkSources_1031716_37.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/52c4a6b0-5455-46b0-8105-c2377b4aab1f_isrForCrosstalkSources_1031716_37.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dcf9537aba6f635c7bd973cd68f7f63 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/26a13790-ec24-4211-904c-06cfb0ed9024_isrForCrosstalkSources_1031712_31.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/26a13790-ec24-4211-904c-06cfb0ed9024_isrForCrosstalkSources_1031712_31.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 483b286bfb1f0f6da98f4e22bf2dd330 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/6ebb85a0-8c0a-4fb0-a91b-ead7390be316_isrForCrosstalkSources_1031705_53.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/6ebb85a0-8c0a-4fb0-a91b-ead7390be316_isrForCrosstalkSources_1031705_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:02:52 proc_lsst.multi:146 [INFO] found job 17147935 in provider astro proc_lsst.multi INFO: found job 17147935 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:02:53 proc_lsst.multi:146 [INFO] found job 17147936 in provider astro proc_lsst.multi INFO: found job 17147936 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:02:54 proc_lsst.multi:146 [INFO] found job 17147937 in provider astro proc_lsst.multi INFO: found job 17147937 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f189d86246c7f425f30ac8c5b4ee82f with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/84575da9-6d89-46db-88f4-b6a26b3b09de_isrForCrosstalkSources_1031736_26.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/84575da9-6d89-46db-88f4-b6a26b3b09de_isrForCrosstalkSources_1031736_26.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a23a1657db3b5ac0dc2b92974762a9b with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/68aee9ea-c24d-4040-ab31-cfd2f8ac3edc_isrForCrosstalkSources_1031721_34.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/68aee9ea-c24d-4040-ab31-cfd2f8ac3edc_isrForCrosstalkSources_1031721_34.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87834c9da1b84408dd656a54a4fa46b6 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f17bd0c7-9767-4a72-a11e-f0592e8ead29_isrForCrosstalkSources_1031735_40.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f17bd0c7-9767-4a72-a11e-f0592e8ead29_isrForCrosstalkSources_1031735_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:02:55 proc_lsst.multi:146 [INFO] found job 17147938 in provider astro proc_lsst.multi INFO: found job 17147938 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06b3ba05a06663952d528bc04a3958c8 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1b827745-2e94-4777-8e51-e963e770e523_isrForCrosstalkSources_1031700_22.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1b827745-2e94-4777-8e51-e963e770e523_isrForCrosstalkSources_1031700_22.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1eea86f8229babe4784b059bb63e9694 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/176fc966-eee8-4f62-ad08-77f3d093c5bd_isrForCrosstalkSources_1031726_2.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/176fc966-eee8-4f62-ad08-77f3d093c5bd_isrForCrosstalkSources_1031726_2.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7033933d5c27307af5625cf5f6a99ca4 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/ca6b8fcc-0a40-4c06-80d7-f283656f4071_isrForCrosstalkSources_1031716_40.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/ca6b8fcc-0a40-4c06-80d7-f283656f4071_isrForCrosstalkSources_1031716_40.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d60329877d4a4f1209e23c74937fafd8 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4f3600a9-5794-4c43-b7af-c45207c8dea4_isrForCrosstalkSources_1031733_28.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4f3600a9-5794-4c43-b7af-c45207c8dea4_isrForCrosstalkSources_1031733_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:02:56 proc_lsst.multi:146 [INFO] found job 17147939 in provider astro proc_lsst.multi INFO: found job 17147939 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89e7362bad54105a72e83bed9b25177a with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/765ce3ef-9192-44b2-aff4-54f1ef2a7b50_isrForCrosstalkSources_1031726_52.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/765ce3ef-9192-44b2-aff4-54f1ef2a7b50_isrForCrosstalkSources_1031726_52.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef1d7bc5b5c7cf60b5da7ca6f2326b1c with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/13d0dd3f-6f20-4de8-91a4-396785d711b0_isrForCrosstalkSources_1031733_12.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/13d0dd3f-6f20-4de8-91a4-396785d711b0_isrForCrosstalkSources_1031733_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:02:58 proc_lsst.multi:146 [INFO] found job 17147940 in provider astro proc_lsst.multi INFO: found job 17147940 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f589137e52d19aba937913a8fca51af with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/2a7384d5-4ca0-443b-b331-ed67eb4c5f11_isrForCrosstalkSources_1031714_3.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/2a7384d5-4ca0-443b-b331-ed67eb4c5f11_isrForCrosstalkSources_1031714_3.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5a62f22cd38bbfe1b138b843f337af6 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/2795895c-8c9f-49b2-9a58-3d6ba9135353_isrForCrosstalkSources_1031738_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/2795895c-8c9f-49b2-9a58-3d6ba9135353_isrForCrosstalkSources_1031738_2.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7f51d97f08ac0a3e69ba2c7b0bfdc02 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/efd0f4ac-9a1e-427c-978f-80592b102ce5_isrForCrosstalkSources_1031705_3.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/efd0f4ac-9a1e-427c-978f-80592b102ce5_isrForCrosstalkSources_1031705_3.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 116863de96532ab1b3ace7f9832e6545 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e772b3fc-e666-4d85-ab79-7aaee698d466_isrForCrosstalkSources_1031707_19.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e772b3fc-e666-4d85-ab79-7aaee698d466_isrForCrosstalkSources_1031707_19.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1844e576f7dbc1607d82b127897cbfc9 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/9e65467a-225d-45dc-8003-27116bfb122d_isrForCrosstalkSources_1031727_19.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/9e65467a-225d-45dc-8003-27116bfb122d_isrForCrosstalkSources_1031727_19.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da96c3040e011be509255fd10300fea7 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fdc12190-7817-46fd-b8af-865e969a28c1_isrForCrosstalkSources_1031700_42.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fdc12190-7817-46fd-b8af-865e969a28c1_isrForCrosstalkSources_1031700_42.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6403d76e8dfa56eefe6178d5a983af82 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6bd867a4-eb1a-433a-9b9c-ce961ad3e6f5_isrForCrosstalkSources_1031728_31.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6bd867a4-eb1a-433a-9b9c-ce961ad3e6f5_isrForCrosstalkSources_1031728_31.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51048fd31cc1c46351b082296d3ffc8c with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c17fe010-0f34-4b79-ad6f-2252b761e4d3_isrForCrosstalkSources_1031716_59.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c17fe010-0f34-4b79-ad6f-2252b761e4d3_isrForCrosstalkSources_1031716_59.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ac06d0fa1471e898eeee21575f1d1ce with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/349fbca3-9f48-4425-a3ec-2eaa954f8006_isrForCrosstalkSources_1031722_10.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/349fbca3-9f48-4425-a3ec-2eaa954f8006_isrForCrosstalkSources_1031722_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:02:59 proc_lsst.multi:146 [INFO] found job 17147941 in provider astro proc_lsst.multi INFO: found job 17147941 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b937ebee0a3a4c9555bfea0a300a6564 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/66b1e396-c38c-4737-a532-1cf985952736_isrForCrosstalkSources_1031712_7.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/66b1e396-c38c-4737-a532-1cf985952736_isrForCrosstalkSources_1031712_7.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c62324202a0574b5f975ac29c14aa1b3 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/f43382ff-d5b6-456a-b303-b07c18df359e_isrForCrosstalkSources_1031718_20.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/f43382ff-d5b6-456a-b303-b07c18df359e_isrForCrosstalkSources_1031718_20.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84cc972fb59e0ce6cf43b7c5108471f0 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a0500995-2190-4aba-b168-378f1f5eeadb_isrForCrosstalkSources_1031720_58.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a0500995-2190-4aba-b168-378f1f5eeadb_isrForCrosstalkSources_1031720_58.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b430e3c26d937515085c9eb8986c09ff with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/304edd5d-bebd-4aa5-b310-ae0c88c40bc9_isrForCrosstalkSources_1031704_49.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/304edd5d-bebd-4aa5-b310-ae0c88c40bc9_isrForCrosstalkSources_1031704_49.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bf2a05fcdb2a3a8fc8de0cb4d7e6627 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ee21d936-a0a6-4c09-92b8-bd07b7e98082_isrForCrosstalkSources_1031737_2.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ee21d936-a0a6-4c09-92b8-bd07b7e98082_isrForCrosstalkSources_1031737_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:02 proc_lsst.multi:146 [INFO] found job 17147942 in provider astro proc_lsst.multi INFO: found job 17147942 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0e47625a6ad0a99d3b901a740cb7f24 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/dac248eb-bbe4-481f-9745-b51c46c31ff8_isrForCrosstalkSources_1031708_3.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/dac248eb-bbe4-481f-9745-b51c46c31ff8_isrForCrosstalkSources_1031708_3.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d0e970bace5bde2bf936302f298c802 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e1881e81-6036-4a7c-8667-4f5a50092cdc_isrForCrosstalkSources_1031725_52.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e1881e81-6036-4a7c-8667-4f5a50092cdc_isrForCrosstalkSources_1031725_52.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b956e03d1e87f31b9cd533f997aac7a with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e1238eab-0a04-4858-9eee-897dbbb5be5a_isrForCrosstalkSources_1031710_25.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e1238eab-0a04-4858-9eee-897dbbb5be5a_isrForCrosstalkSources_1031710_25.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a4a08c4ee312243325b6e578ca00631 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/850714ec-3520-40a6-8c01-a0119fed5c02_isrForCrosstalkSources_1031733_32.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/850714ec-3520-40a6-8c01-a0119fed5c02_isrForCrosstalkSources_1031733_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:05 proc_lsst.multi:146 [INFO] found job 17147943 in provider astro proc_lsst.multi INFO: found job 17147943 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee4948d189d226ba9c38bdd3ba2fee3f with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/80ecf0e6-fc37-41e4-a9d5-91ed301731e8_isrForCrosstalkSources_1031720_57.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/80ecf0e6-fc37-41e4-a9d5-91ed301731e8_isrForCrosstalkSources_1031720_57.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a2c2005f8f00926e9543b339be04faf with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/171647c4-75ce-4f6c-9735-fbc0f4d765df_isrForCrosstalkSources_1031730_3.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/171647c4-75ce-4f6c-9735-fbc0f4d765df_isrForCrosstalkSources_1031730_3.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d9fa4e4a72a313fdd14ebb906e70269 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/e01ea1f7-84bd-4082-b7fa-2a442a699cf8_isrForCrosstalkSources_1031699_15.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/e01ea1f7-84bd-4082-b7fa-2a442a699cf8_isrForCrosstalkSources_1031699_15.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db6967f4f7cfdeec1d3d1725ec10af01 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fdd2240b-f1e0-41b7-af8a-bfc3e4e4f33b_isrForCrosstalkSources_1031732_47.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fdd2240b-f1e0-41b7-af8a-bfc3e4e4f33b_isrForCrosstalkSources_1031732_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:06 proc_lsst.multi:146 [INFO] found job 17147944 in provider astro proc_lsst.multi INFO: found job 17147944 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64d0234798ef8e714d1b00f3bd0c9d1e with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/51135140-f408-49cd-a8f7-2a4fd2863291_isrForCrosstalkSources_1031702_52.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/51135140-f408-49cd-a8f7-2a4fd2863291_isrForCrosstalkSources_1031702_52.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6a87726dd647406d0162fbbbe423de2 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/4c517d0f-5a9d-4bf6-b145-032fecf4e16a_isrForCrosstalkSources_1031722_54.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/4c517d0f-5a9d-4bf6-b145-032fecf4e16a_isrForCrosstalkSources_1031722_54.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ead527d4a749412f9216159d44f5fb38 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/93689b0f-907f-42b6-9418-e7ef72530b86_isrForCrosstalkSources_1031730_22.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/93689b0f-907f-42b6-9418-e7ef72530b86_isrForCrosstalkSources_1031730_22.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fca282816f3cfc05e34a41708a9e1ff7 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/73b24f5c-1014-4655-87be-aa2a91144e5b_isrForCrosstalkSources_1031719_54.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/73b24f5c-1014-4655-87be-aa2a91144e5b_isrForCrosstalkSources_1031719_54.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fd4976132de0f041e4b7d718bccbe20 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4145b74f-0991-4fa6-bf19-9ddc53e163d3_isrForCrosstalkSources_1031715_57.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4145b74f-0991-4fa6-bf19-9ddc53e163d3_isrForCrosstalkSources_1031715_57.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2c0f61f9e6fd62e9351700fe023aa57 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/03808966-6af9-4840-988e-1367e25abf52_isrForCrosstalkSources_1031730_14.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/03808966-6af9-4840-988e-1367e25abf52_isrForCrosstalkSources_1031730_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:08 proc_lsst.multi:146 [INFO] found job 17147945 in provider astro proc_lsst.multi INFO: found job 17147945 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af5eec94e2f538a96614a239838d7aae with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/6b8f6538-0662-435c-8f22-239c3c95c64d_isrForCrosstalkSources_1031725_14.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/6b8f6538-0662-435c-8f22-239c3c95c64d_isrForCrosstalkSources_1031725_14.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73d8c5304fe6a8fb992d4326384e8b69 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/ff14a77b-a283-42e5-a04e-bda20e7269e5_isrForCrosstalkSources_1031735_45.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/ff14a77b-a283-42e5-a04e-bda20e7269e5_isrForCrosstalkSources_1031735_45.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b23a81cbe3c5f92c5d96208fb80de591 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/82b2ba86-91c2-4f53-b030-eb1005d62696_isrForCrosstalkSources_1031712_46.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/82b2ba86-91c2-4f53-b030-eb1005d62696_isrForCrosstalkSources_1031712_46.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ec4dba906b77d1f23db86dfdbc01289 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/3ac2128f-4874-44ea-88ae-f837263099f7_isrForCrosstalkSources_1031704_51.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/3ac2128f-4874-44ea-88ae-f837263099f7_isrForCrosstalkSources_1031704_51.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df4cb016ed6fdbd0c88d18663ce77392 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/26f2aa33-df17-4d62-be64-b951768635ff_isrForCrosstalkSources_1031710_59.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/26f2aa33-df17-4d62-be64-b951768635ff_isrForCrosstalkSources_1031710_59.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e8187aeb0c1c5d7e84bcf580677c905 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/26b2b291-9c80-484f-98b7-8e622270fecd_isrForCrosstalkSources_1031722_37.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/26b2b291-9c80-484f-98b7-8e622270fecd_isrForCrosstalkSources_1031722_37.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf1e3f989304e7c17a2a60baf32d2bc4 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a014b8b0-3760-4e8a-aa32-d938b1ea0cd0_isrForCrosstalkSources_1031728_30.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a014b8b0-3760-4e8a-aa32-d938b1ea0cd0_isrForCrosstalkSources_1031728_30.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ba4365831edb54978af767cad5ef1cb with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cacb828e-c2ec-4518-847e-cb1fa69a97fa_isrForCrosstalkSources_1031728_61.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cacb828e-c2ec-4518-847e-cb1fa69a97fa_isrForCrosstalkSources_1031728_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:10 proc_lsst.multi:146 [INFO] found job 17147946 in provider astro proc_lsst.multi INFO: found job 17147946 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.memoization DEBUG: Storing app cache entry 1f22338f0b2ff6f17d6c55d6aca29dba with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/53370d68-ba9e-47cc-a669-ff40561222e4_isrForCrosstalkSources_1031732_32.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/53370d68-ba9e-47cc-a669-ff40561222e4_isrForCrosstalkSources_1031732_32.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1869453cc6e99f6ee41f0222a66c1bf2 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4b3a4136-8834-4b70-8c25-6e28f4e8c5e0_isrForCrosstalkSources_1031716_21.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4b3a4136-8834-4b70-8c25-6e28f4e8c5e0_isrForCrosstalkSources_1031716_21.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1827bdd27496b8e432a6f55d62153edb with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6ca16a7b-685e-4c23-a4ad-4abe942ec01d_isrForCrosstalkSources_1031711_35.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6ca16a7b-685e-4c23-a4ad-4abe942ec01d_isrForCrosstalkSources_1031711_35.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d484636e0a657045b2de4b513dd5cc0 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/e7606d77-3625-4973-9b89-f62ebf87d594_isrForCrosstalkSources_1031721_46.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/e7606d77-3625-4973-9b89-f62ebf87d594_isrForCrosstalkSources_1031721_46.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3371386bbf2ee46a145bc6d19b578bcf with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ad1784b4-c7d0-40b3-8431-2f70d865bb8f_isrForCrosstalkSources_1031717_42.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ad1784b4-c7d0-40b3-8431-2f70d865bb8f_isrForCrosstalkSources_1031717_42.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb194b150e62a7c70545d3f930f1356b with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0fc8cce6-251e-482d-9e61-968f4c3a1fef_isrForCrosstalkSources_1031717_6.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0fc8cce6-251e-482d-9e61-968f4c3a1fef_isrForCrosstalkSources_1031717_6.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b567c3df9b749427a181aad6e140fdde with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/63e5c995-8288-495f-927c-5a472840d661_isrForCrosstalkSources_1031701_20.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/63e5c995-8288-495f-927c-5a472840d661_isrForCrosstalkSources_1031701_20.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4fa17d2c65a16ef07980654a52e47c1 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1cd64e75-ee74-45e1-a954-57e1f08cb262_isrForCrosstalkSources_1031713_11.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1cd64e75-ee74-45e1-a954-57e1f08cb262_isrForCrosstalkSources_1031713_11.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bb49e49dc3a402ca130afaeb489a57b with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5fcf73bd-6f7e-4fae-86c3-dd9f0df0c3c9_isrForCrosstalkSources_1031724_47.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5fcf73bd-6f7e-4fae-86c3-dd9f0df0c3c9_isrForCrosstalkSources_1031724_47.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09864999fafb658a7a7642ff95e5d973 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2e54a598-3267-4d82-a00d-1dd0da717540_isrForCrosstalkSources_1031704_53.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2e54a598-3267-4d82-a00d-1dd0da717540_isrForCrosstalkSources_1031704_53.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cd730b6d26cc562343b2604c208388b with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fdfca7c1-f1f4-4dbf-a226-38ad7c165351_isrForCrosstalkSources_1031723_33.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fdfca7c1-f1f4-4dbf-a226-38ad7c165351_isrForCrosstalkSources_1031723_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:12 proc_lsst.multi:146 [INFO] found job 17147947 in provider astro proc_lsst.multi INFO: found job 17147947 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 105ad82e9363f71617b63f6f2d6d67f7 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/4e2f7527-d01a-4f39-b624-3c5371bd67af_isrForCrosstalkSources_1031710_5.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/4e2f7527-d01a-4f39-b624-3c5371bd67af_isrForCrosstalkSources_1031710_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:14 proc_lsst.multi:146 [INFO] found job 17147948 in provider astro proc_lsst.multi INFO: found job 17147948 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b352c3cac5a051e6692eb018df5f141 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/81f71d40-aaf8-4568-9f8c-31138440d617_isrForCrosstalkSources_1031727_8.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/81f71d40-aaf8-4568-9f8c-31138440d617_isrForCrosstalkSources_1031727_8.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 822f79575defae3b792f43d989ad702a with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/00103313-056a-46d5-8b04-3fac4fc70ad9_isrForCrosstalkSources_1031706_40.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/00103313-056a-46d5-8b04-3fac4fc70ad9_isrForCrosstalkSources_1031706_40.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7594ecfbed93790c1a91c44f0493968 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/0d23a2e7-80ba-4a08-85f1-ffb5c528f140_isrForCrosstalkSources_1031700_18.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/0d23a2e7-80ba-4a08-85f1-ffb5c528f140_isrForCrosstalkSources_1031700_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:16 proc_lsst.multi:146 [INFO] found job 17147950 in provider astro proc_lsst.multi INFO: found job 17147950 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 949067ceb84409d89254d52420792b32 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/db1935b7-b725-40df-99c2-2153cb95da85_isrForCrosstalkSources_1031700_7.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/db1935b7-b725-40df-99c2-2153cb95da85_isrForCrosstalkSources_1031700_7.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 443004bb6aac2cf3777aaee7244b0bb2 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c89fc29c-3da9-46dd-b72a-ab9fef044a6a_isrForCrosstalkSources_1031710_13.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c89fc29c-3da9-46dd-b72a-ab9fef044a6a_isrForCrosstalkSources_1031710_13.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7998d1f56610e45d3105a5351526b92 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4c8cdd8f-95c3-4f7b-acba-12adf2a9058a_isrForCrosstalkSources_1031737_42.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4c8cdd8f-95c3-4f7b-acba-12adf2a9058a_isrForCrosstalkSources_1031737_42.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8fa685b280558eef8d083b52dff38be with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/af153a57-ec3f-44a7-b1f0-167504addc1d_isrForCrosstalkSources_1031715_41.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/af153a57-ec3f-44a7-b1f0-167504addc1d_isrForCrosstalkSources_1031715_41.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cdf63d5921cf02f4347dd90f4bc79ef with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/55f9357d-bc22-4d82-b810-0d755189d567_isrForCrosstalkSources_1031737_31.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/55f9357d-bc22-4d82-b810-0d755189d567_isrForCrosstalkSources_1031737_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:17 proc_lsst.multi:146 [INFO] found job 17147951 in provider astro proc_lsst.multi INFO: found job 17147951 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 786c49dd889b680272a77d4e7b0738cb with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/8af6e640-f46e-4495-8f53-99c2b668dce4_isrForCrosstalkSources_1031737_33.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/8af6e640-f46e-4495-8f53-99c2b668dce4_isrForCrosstalkSources_1031737_33.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3db81ea76b43d709d63aec9816c8050a with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/61147069-14fe-4957-9fc2-de548f82b6cf_isrForCrosstalkSources_1031716_23.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/61147069-14fe-4957-9fc2-de548f82b6cf_isrForCrosstalkSources_1031716_23.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fb6f8d029494c8b1a9d848e2aade360 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/86727ae2-a631-4a68-8534-3ad7e026ee54_isrForCrosstalkSources_1031732_20.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/86727ae2-a631-4a68-8534-3ad7e026ee54_isrForCrosstalkSources_1031732_20.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65fa9b168462567a96fe2218c841506c with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/89e71966-c38c-4635-ab56-5c59d29dcfc6_isrForCrosstalkSources_1031722_41.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/89e71966-c38c-4635-ab56-5c59d29dcfc6_isrForCrosstalkSources_1031722_41.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63e462f9f8b6b253ce8025e15ba391e6 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2e2c2e1e-ea2f-4f9e-b9bb-2868491444bd_isrForCrosstalkSources_1031734_27.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2e2c2e1e-ea2f-4f9e-b9bb-2868491444bd_isrForCrosstalkSources_1031734_27.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3c3b684f5c6b6a15ddcb153a6dbcb68 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/99c33805-eef5-4ea5-b331-139f7f2acc6c_isrForCrosstalkSources_1031700_17.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/99c33805-eef5-4ea5-b331-139f7f2acc6c_isrForCrosstalkSources_1031700_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:20 proc_lsst.multi:146 [INFO] found job 17147952 in provider astro proc_lsst.multi INFO: found job 17147952 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e947da3c83dbb359c6ba3a4d3ab9325f with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/ec92e6c3-7cad-4f5a-a861-baaf0f1ec6b9_isrForCrosstalkSources_1031723_27.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/ec92e6c3-7cad-4f5a-a861-baaf0f1ec6b9_isrForCrosstalkSources_1031723_27.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28656823fac51e2d7ce3949f68a4eaaf with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/92f4fb91-ffbf-4047-96c5-2926aa607f2a_isrForCrosstalkSources_1031718_37.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/92f4fb91-ffbf-4047-96c5-2926aa607f2a_isrForCrosstalkSources_1031718_37.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3328d69ebfad71ccc62a0cb36ee5fbf with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/d7b87ed9-b1ec-4a8a-b2a0-7cc0ac22ad35_isrForCrosstalkSources_1031710_1.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/d7b87ed9-b1ec-4a8a-b2a0-7cc0ac22ad35_isrForCrosstalkSources_1031710_1.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b469416942e382a87cb5cc3b60476fe3 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/a15ccee3-a972-47b2-80cc-993dca69a3fe_isrForCrosstalkSources_1031710_37.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/a15ccee3-a972-47b2-80cc-993dca69a3fe_isrForCrosstalkSources_1031710_37.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5516ce05eb8ae11e0fbadec260f6275 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/01df9b3d-ab9f-4b27-954b-40ce5c93c868_isrForCrosstalkSources_1031725_43.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/01df9b3d-ab9f-4b27-954b-40ce5c93c868_isrForCrosstalkSources_1031725_43.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d997f45a984c15610ab75083bd3a107e with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d908bf00-fe86-4e32-8082-2b9514896ffd_isrForCrosstalkSources_1031737_10.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d908bf00-fe86-4e32-8082-2b9514896ffd_isrForCrosstalkSources_1031737_10.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e8ec9b55da21c821622df98b3516958 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/5f6aef4e-69ec-4b25-97ff-a111e319c878_isrForCrosstalkSources_1031707_6.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/5f6aef4e-69ec-4b25-97ff-a111e319c878_isrForCrosstalkSources_1031707_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:22 proc_lsst.multi:146 [INFO] found job 17147953 in provider astro proc_lsst.multi INFO: found job 17147953 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0faf8d91854ed607c19653ba9e1ec249 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d12a0f7f-90c0-49aa-a90e-22757ab9b469_isrForCrosstalkSources_1031703_24.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d12a0f7f-90c0-49aa-a90e-22757ab9b469_isrForCrosstalkSources_1031703_24.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21f95e25ada5e1793998a17f2a7a9ef5 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4a9dca73-0ca0-47da-971e-e809614442ec_isrForCrosstalkSources_1031732_56.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4a9dca73-0ca0-47da-971e-e809614442ec_isrForCrosstalkSources_1031732_56.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ba41316ece65e2298e7e1d01f8a3385 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4feb177d-73b6-45ca-876d-4999ee0fd38f_isrForCrosstalkSources_1031733_25.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4feb177d-73b6-45ca-876d-4999ee0fd38f_isrForCrosstalkSources_1031733_25.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70987c838a5d0e854647b3b5d2a5a0ff with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/eebe3a0a-a2bd-4c5c-b34e-5c7df383bebd_isrForCrosstalkSources_1031701_13.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/eebe3a0a-a2bd-4c5c-b34e-5c7df383bebd_isrForCrosstalkSources_1031701_13.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e539bcb00b3dc86f4661083d14c128b with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/b8bf4923-5c06-48c9-9b61-d355b35ffb61_isrForCrosstalkSources_1031710_51.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/b8bf4923-5c06-48c9-9b61-d355b35ffb61_isrForCrosstalkSources_1031710_51.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c32d4d189d51bc6f128970664ca14064 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8c8e3ecf-ddf8-4433-82ad-73820ce8401e_isrForCrosstalkSources_1031721_1.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8c8e3ecf-ddf8-4433-82ad-73820ce8401e_isrForCrosstalkSources_1031721_1.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1509d8f03f1dda4e8d5bccb758cf57 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2870c680-dda5-4dd5-912b-fb1df87972dc_isrForCrosstalkSources_1031727_60.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2870c680-dda5-4dd5-912b-fb1df87972dc_isrForCrosstalkSources_1031727_60.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e7121151c00cc1968cacfb73bc6011b with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/6c0da8a4-a066-4ba8-bf3a-5d2023fcc470_isrForCrosstalkSources_1031713_28.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/6c0da8a4-a066-4ba8-bf3a-5d2023fcc470_isrForCrosstalkSources_1031713_28.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38d9ad53b1258552f0c41cc8fcbe48aa with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0df69710-4383-490c-83a1-abdcbd6cf99b_isrForCrosstalkSources_1031720_13.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0df69710-4383-490c-83a1-abdcbd6cf99b_isrForCrosstalkSources_1031720_13.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf21ff80ca0b034e38723ff67233f3dd with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8177bfa3-003c-4673-ac27-fe26c9706e01_isrForCrosstalkSources_1031699_42.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8177bfa3-003c-4673-ac27-fe26c9706e01_isrForCrosstalkSources_1031699_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:25 proc_lsst.multi:146 [INFO] found job 17147954 in provider astro proc_lsst.multi INFO: found job 17147954 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96e696583587878426548218ea7683b9 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/538809e8-303e-4f81-a6af-41c1775b45a6_isrForCrosstalkSources_1031726_57.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/538809e8-303e-4f81-a6af-41c1775b45a6_isrForCrosstalkSources_1031726_57.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9093b72ffc3e0527f8736f8d7c6f7df4 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/470c4db5-36a0-48ab-a504-e1fe6980bc6a_isrForCrosstalkSources_1031735_32.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/470c4db5-36a0-48ab-a504-e1fe6980bc6a_isrForCrosstalkSources_1031735_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:26 proc_lsst.multi:146 [INFO] found job 17147955 in provider astro proc_lsst.multi INFO: found job 17147955 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:27 proc_lsst.multi:146 [INFO] found job 17147956 in provider astro proc_lsst.multi INFO: found job 17147956 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fb05abf0733ab3f13adc0722360bc06 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7220336c-3444-4ada-8b91-50ba0af32b00_isrForCrosstalkSources_1031714_11.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7220336c-3444-4ada-8b91-50ba0af32b00_isrForCrosstalkSources_1031714_11.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e9a0f795081aa5f3268319288240954 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/75dae491-2bb8-45bb-8cc7-3952d35b53f6_isrForCrosstalkSources_1031717_35.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/75dae491-2bb8-45bb-8cc7-3952d35b53f6_isrForCrosstalkSources_1031717_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:28 proc_lsst.multi:146 [INFO] found job 17147957 in provider astro proc_lsst.multi INFO: found job 17147957 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:29 proc_lsst.multi:146 [INFO] found job 17147958 in provider astro proc_lsst.multi INFO: found job 17147958 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cabe8ae05461ef281401bc7293d9c963 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/fce6dc21-8290-4444-ad41-6f6d113affab_isrForCrosstalkSources_1031703_56.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/fce6dc21-8290-4444-ad41-6f6d113affab_isrForCrosstalkSources_1031703_56.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2d11a82d97ee9295897ae666bb1364d with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/83f0c4b6-f814-4b0f-969f-ab6ccdcee944_isrForCrosstalkSources_1031724_30.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/83f0c4b6-f814-4b0f-969f-ab6ccdcee944_isrForCrosstalkSources_1031724_30.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd3bbfec0e94c23118cc43d8be1ff963 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9c6072fe-121b-437f-ab9e-757008e9a5df_isrForCrosstalkSources_1031730_17.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9c6072fe-121b-437f-ab9e-757008e9a5df_isrForCrosstalkSources_1031730_17.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed9a38c72af0f800550aa59e2aa5f8f2 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/24b779b6-ebc9-423e-a258-297317e5afd3_isrForCrosstalkSources_1031700_35.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/24b779b6-ebc9-423e-a258-297317e5afd3_isrForCrosstalkSources_1031700_35.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38aaca494ba43f83986079c5162e17f5 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eab7f269-99d5-46b9-bf3b-6500ca062c03_isrForCrosstalkSources_1031719_23.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eab7f269-99d5-46b9-bf3b-6500ca062c03_isrForCrosstalkSources_1031719_23.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3236da852ce4b6871f61807c72c79adb with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/74c916ed-9d80-4035-9464-683a551f8052_isrForCrosstalkSources_1031737_51.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/74c916ed-9d80-4035-9464-683a551f8052_isrForCrosstalkSources_1031737_51.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a14268a5c83a8fb92b7490124b7c8cc1 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cee2b533-a441-4b29-bac7-a199759e4892_isrForCrosstalkSources_1031716_24.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cee2b533-a441-4b29-bac7-a199759e4892_isrForCrosstalkSources_1031716_24.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a48684776306d673338f0c428d45a4c with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/1d710df2-71d3-4716-9d89-9e8e3e9b296b_isrForCrosstalkSources_1031716_32.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/1d710df2-71d3-4716-9d89-9e8e3e9b296b_isrForCrosstalkSources_1031716_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:30 proc_lsst.multi:146 [INFO] found job 17147959 in provider astro proc_lsst.multi INFO: found job 17147959 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f2abfeca08626b73f237de1241ee19c with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e54e5775-45a6-4a10-a30d-3499abee317b_isrForCrosstalkSources_1031712_58.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e54e5775-45a6-4a10-a30d-3499abee317b_isrForCrosstalkSources_1031712_58.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 746d0a9b59fd561b10ea39ab28782d52 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/bfcc5f02-8124-4815-8fc4-4ec077c08156_isrForCrosstalkSources_1031726_21.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/bfcc5f02-8124-4815-8fc4-4ec077c08156_isrForCrosstalkSources_1031726_21.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2accdbc85c3c9a6622c1e5307840fa4 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ddcb73e5-b00f-4de0-a2dc-fd5743b892d6_isrForCrosstalkSources_1031713_6.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ddcb73e5-b00f-4de0-a2dc-fd5743b892d6_isrForCrosstalkSources_1031713_6.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 739aa4c6b3ac44ef364bf204d8c67a81 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/66b046f1-b983-44e3-8d6a-cc4d816bdaf2_isrForCrosstalkSources_1031715_24.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/66b046f1-b983-44e3-8d6a-cc4d816bdaf2_isrForCrosstalkSources_1031715_24.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a097f6a091aaf7b1ffc1ba121a41ae9f with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4133ad94-6d03-491c-9ae7-f6cf77b01950_isrForCrosstalkSources_1031709_12.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4133ad94-6d03-491c-9ae7-f6cf77b01950_isrForCrosstalkSources_1031709_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:32 proc_lsst.multi:146 [INFO] found job 17147960 in provider astro proc_lsst.multi INFO: found job 17147960 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 287be13cc92bb8869bf71e3d6ad84bd3 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/927cf93e-5d16-4dab-a6a7-6f74e2fbf47f_isrForCrosstalkSources_1031725_35.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/927cf93e-5d16-4dab-a6a7-6f74e2fbf47f_isrForCrosstalkSources_1031725_35.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed918a4426b81ffac6c95082b24d270a with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c0360758-26fa-454a-ab12-3cf0b53d1758_isrForCrosstalkSources_1031703_52.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c0360758-26fa-454a-ab12-3cf0b53d1758_isrForCrosstalkSources_1031703_52.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29d66fcdb86ac84e5d0da985b868b7eb with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/44a287b7-6f07-4451-a858-92a51ac71f5f_isrForCrosstalkSources_1031712_42.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/44a287b7-6f07-4451-a858-92a51ac71f5f_isrForCrosstalkSources_1031712_42.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 784a148fcf9d31c73ca172e2e3cd4960 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/66ed846e-dbec-4454-b085-d84fdefe1f8f_isrForCrosstalkSources_1031714_50.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/66ed846e-dbec-4454-b085-d84fdefe1f8f_isrForCrosstalkSources_1031714_50.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 247ccccf0d615391560f8e10ff8767e7 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d2cada02-4be5-4563-96e1-7a6d461dd6a9_isrForCrosstalkSources_1031704_31.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d2cada02-4be5-4563-96e1-7a6d461dd6a9_isrForCrosstalkSources_1031704_31.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6212f50a59fe74eb193fd00b0df787b7 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/63837bd4-4315-47cf-bcda-827bb5c3839b_isrForCrosstalkSources_1031731_18.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/63837bd4-4315-47cf-bcda-827bb5c3839b_isrForCrosstalkSources_1031731_18.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 307b5f14f04b05077050eabbcf00e713 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a79b5650-d0d7-4671-8bc2-2902565b4e7d_isrForCrosstalkSources_1031721_51.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a79b5650-d0d7-4671-8bc2-2902565b4e7d_isrForCrosstalkSources_1031721_51.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 622b583a1d5a700950b173147d7c0072 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fb91c96f-9fab-4107-b950-f50b4680799c_isrForCrosstalkSources_1031702_3.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fb91c96f-9fab-4107-b950-f50b4680799c_isrForCrosstalkSources_1031702_3.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e7baa47054ff053d21c92ac1b8817ef with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/c66dadcd-fe6d-4363-8a1e-eec49b8b26fe_isrForCrosstalkSources_1031721_3.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/c66dadcd-fe6d-4363-8a1e-eec49b8b26fe_isrForCrosstalkSources_1031721_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:36 proc_lsst.multi:146 [INFO] found job 17147961 in provider astro proc_lsst.multi INFO: found job 17147961 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147931,17147932,17147933,17147934,17147935,17147936,17147937,17147938,17147939,17147940,17147941,17147942,17147943,17147944,17147945,17147946,17147947,17147948,17147950,17147951,17147952,17147953,17147954,17147955,17147956,17147957,17147958,17147959,17147960,17147961' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ed0d350001831cfe7755603ce86a9a5 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/379e3d6b-bea4-414e-aac7-a7b39c03f7a7_isrForCrosstalkSources_1031738_31.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/379e3d6b-bea4-414e-aac7-a7b39c03f7a7_isrForCrosstalkSources_1031738_31.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5c8f029fdda42d6a5e04783515ce26d with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/fb21f9ff-918d-46a2-8ebf-1e62496a2cf1_isrForCrosstalkSources_1031708_18.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/fb21f9ff-918d-46a2-8ebf-1e62496a2cf1_isrForCrosstalkSources_1031708_18.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 214beafbf1a6a97112fa1b58eb31f4d9 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/5b5b9e2c-6fa6-4832-b01f-3cc93d86e09a_isrForCrosstalkSources_1031729_41.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/5b5b9e2c-6fa6-4832-b01f-3cc93d86e09a_isrForCrosstalkSources_1031729_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147961 R 17147960 R 17147959 R 17147958 R 17147957 R 17147955 R 17147956 R 17147952 R 17147953 R 17147954 R 17147950 R 17147951 R 17147948 R 17147946 R 17147947 R 17147943 R 17147944 R 17147945 R 17147938 R 17147939 R 17147940 R 17147941 R 17147942 R 17147934 R 17147935 R 17147936 R 17147937 R 17147931 R 17147932 R 17147933 R parsl.providers.slurm.slurm DEBUG: Updating job 17147961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147948 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147946 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147947 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147943 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147944 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147945 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147938 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147939 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147940 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147941 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147942 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147935 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147936 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17147937 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147931 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147932 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147933 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:37 proc_lsst.multi:146 [INFO] found job 17147962 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147962 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147962,17147963,17147964,17147965,17147966' parsl.dataflow.memoization DEBUG: Storing app cache entry 4aa0ee2f1679572c0de910e6a62e949f with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/78af5620-ea33-4ecf-b8b4-36666604ebab_isrForCrosstalkSources_1031718_57.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/78af5620-ea33-4ecf-b8b4-36666604ebab_isrForCrosstalkSources_1031718_57.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f18c5814dd2b83a0ed19737ba5a34af4 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ded29da8-f260-4c29-bba7-26558c24c08a_isrForCrosstalkSources_1031717_58.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ded29da8-f260-4c29-bba7-26558c24c08a_isrForCrosstalkSources_1031717_58.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2037a4eb36b5cd5ca029d343035f965d with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/8bd91eae-ebfe-4c16-be47-6610a6c5612e_isrForCrosstalkSources_1031726_62.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/8bd91eae-ebfe-4c16-be47-6610a6c5612e_isrForCrosstalkSources_1031726_62.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c2a4c70db9521b368c665ecd5a4804e with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/496ef789-a5c0-46e2-b10c-ee15555d8af8_isrForCrosstalkSources_1031732_54.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/496ef789-a5c0-46e2-b10c-ee15555d8af8_isrForCrosstalkSources_1031732_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147966 R 17147964 R 17147965 R 17147963 R 17147962 R parsl.providers.slurm.slurm DEBUG: Updating job 17147966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147962 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:38 proc_lsst.multi:146 [INFO] found job 17147963 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147963 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147962,17147963,17147964,17147965,17147966' parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc683686fb762c34d57e9efc29772426 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/aef745b4-48d1-46cc-8c2c-aaa2d115eba4_isrForCrosstalkSources_1031720_29.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/aef745b4-48d1-46cc-8c2c-aaa2d115eba4_isrForCrosstalkSources_1031720_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147966 R 17147964 R 17147965 R 17147963 R 17147962 R parsl.providers.slurm.slurm DEBUG: Updating job 17147966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147962 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:40 proc_lsst.multi:146 [INFO] found job 17147964 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147964 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147962,17147963,17147964,17147965,17147966' parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f04358dbc142da2fdb61fc7993d9ebb with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/18d1defe-f7cd-4d2f-a53d-9e572664f8ef_isrForCrosstalkSources_1031717_34.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/18d1defe-f7cd-4d2f-a53d-9e572664f8ef_isrForCrosstalkSources_1031717_34.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c20572bff195fa3946f829247ea7df25 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ed710321-816b-4ddb-9882-6cd19c42d352_isrForCrosstalkSources_1031724_3.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ed710321-816b-4ddb-9882-6cd19c42d352_isrForCrosstalkSources_1031724_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147966 R 17147964 R 17147965 R 17147963 R 17147962 R parsl.providers.slurm.slurm DEBUG: Updating job 17147966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147962 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:41 proc_lsst.multi:146 [INFO] found job 17147965 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147965 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147962,17147963,17147964,17147965,17147966' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11e517559fa6439d0e5c6479baac7bb1 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/82e4b1be-f989-4bc5-8556-d0cc065ca407_isrForCrosstalkSources_1031711_20.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/82e4b1be-f989-4bc5-8556-d0cc065ca407_isrForCrosstalkSources_1031711_20.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf46981e643ad4c8391c28da7b90dbfc with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/265f60f9-4c48-4416-9e0d-f0272bb7e1a4_isrForCrosstalkSources_1031702_33.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/265f60f9-4c48-4416-9e0d-f0272bb7e1a4_isrForCrosstalkSources_1031702_33.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d1f402c8cf31eba922480150278020c with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/499e9489-f425-4e72-8ef9-f7540f950eaa_isrForCrosstalkSources_1031703_4.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/499e9489-f425-4e72-8ef9-f7540f950eaa_isrForCrosstalkSources_1031703_4.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a305c6f1a09ffd8c4765d9d7650b273 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/17806f93-ccca-463d-84fc-8b0a931d505c_isrForCrosstalkSources_1031728_13.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/17806f93-ccca-463d-84fc-8b0a931d505c_isrForCrosstalkSources_1031728_13.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adcc27788c56fe26a88d618743e3fcf8 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5d16131a-64ec-4fd9-a2b7-b02c4a28a374_isrForCrosstalkSources_1031735_39.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5d16131a-64ec-4fd9-a2b7-b02c4a28a374_isrForCrosstalkSources_1031735_39.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd7d0efacd98b968729dd30f1926ee75 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/aee614a6-2f70-4c0b-8b56-c05d5fdfc1cf_isrForCrosstalkSources_1031712_56.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/aee614a6-2f70-4c0b-8b56-c05d5fdfc1cf_isrForCrosstalkSources_1031712_56.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4396d981e3b075070cd9f45c37ecedad with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/6a8f6aac-361e-44dd-9b74-fb4bcd6a5b51_isrForCrosstalkSources_1031704_45.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/6a8f6aac-361e-44dd-9b74-fb4bcd6a5b51_isrForCrosstalkSources_1031704_45.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 388d6fa564fdf698c2af4ae26b7e20b9 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8d784055-7008-453e-88b6-b5479ac1583b_isrForCrosstalkSources_1031732_11.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8d784055-7008-453e-88b6-b5479ac1583b_isrForCrosstalkSources_1031732_11.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c80f0b9627198bec12e18b3ad13e67e8 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/9767639d-95f1-41ef-9e22-779ba07df2b0_isrForCrosstalkSources_1031705_46.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/9767639d-95f1-41ef-9e22-779ba07df2b0_isrForCrosstalkSources_1031705_46.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd3afb9c9f86f9eb43733839848be335 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/fda1bbfb-d0cf-48ab-a7ce-fabbde48dcc2_isrForCrosstalkSources_1031718_9.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/fda1bbfb-d0cf-48ab-a7ce-fabbde48dcc2_isrForCrosstalkSources_1031718_9.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60fff2e0ba407e18758a2cc3f3698b09 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a00ac069-754b-4ce9-b22f-8244985c5b09_isrForCrosstalkSources_1031712_24.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a00ac069-754b-4ce9-b22f-8244985c5b09_isrForCrosstalkSources_1031712_24.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3417f2654e9fe4321ec27a8c753086ae with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2f23ba89-11a2-48fc-bb43-175fd0691a61_isrForCrosstalkSources_1031726_12.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2f23ba89-11a2-48fc-bb43-175fd0691a61_isrForCrosstalkSources_1031726_12.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 063dae4f662e5861a21c2b6f9c2c4db0 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cd37dbe4-9417-4bfd-a699-3b92811f4230_isrForCrosstalkSources_1031707_5.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cd37dbe4-9417-4bfd-a699-3b92811f4230_isrForCrosstalkSources_1031707_5.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd80ae7cb64995e5a65e27aaa9354482 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/1be21c15-6496-422a-9f59-77714aa59121_isrForCrosstalkSources_1031718_26.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/1be21c15-6496-422a-9f59-77714aa59121_isrForCrosstalkSources_1031718_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147966 R 17147964 R 17147965 R 17147963 R 17147962 R parsl.providers.slurm.slurm DEBUG: Updating job 17147966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147962 with slurm status R to parsl state JobState.RUNNING 2024-03-17 04:03:43 proc_lsst.multi:146 [INFO] found job 17147966 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147966 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17147962,17147963,17147964,17147965,17147966' parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c3d999be628aba583dd1757ead600ae with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/840734b7-5f13-4aad-97d5-651117dd2ce1_isrForCrosstalkSources_1031699_32.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/840734b7-5f13-4aad-97d5-651117dd2ce1_isrForCrosstalkSources_1031699_32.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8873eb9a0659f20061b06899cd0b095a with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/0f4ecfa4-866b-480d-a782-2180f83f41fc_isrForCrosstalkSources_1031735_8.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/0f4ecfa4-866b-480d-a782-2180f83f41fc_isrForCrosstalkSources_1031735_8.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d60afbe0dbf317ec556f9c5136c0ede with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/147b730b-556c-40a2-9245-88fed9c43d65_isrForCrosstalkSources_1031705_16.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/147b730b-556c-40a2-9245-88fed9c43d65_isrForCrosstalkSources_1031705_16.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3a01be9bd018afa86d62bcf4fcf97fa with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/55526ad2-26a0-4332-80ae-6c800a5588b8_isrForCrosstalkSources_1031717_38.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/55526ad2-26a0-4332-80ae-6c800a5588b8_isrForCrosstalkSources_1031717_38.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bff9b96507bfcc3ad0b370a6d0a0061 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/978b7c78-9e20-4072-ae3f-3e5485915ebf_isrForCrosstalkSources_1031704_35.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/978b7c78-9e20-4072-ae3f-3e5485915ebf_isrForCrosstalkSources_1031704_35.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15edae87ed83fa21d2bb2254199e2196 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/d07667de-d731-45be-b5c5-578b39569423_isrForCrosstalkSources_1031708_38.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/d07667de-d731-45be-b5c5-578b39569423_isrForCrosstalkSources_1031708_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17147966 R 17147964 R 17147965 R 17147963 R 17147962 R parsl.providers.slurm.slurm DEBUG: Updating job 17147966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17147962 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 36, active_tasks = 2281 parsl.dataflow.strategy DEBUG: Executor multi has 2281 active tasks, 36/0 running/pending blocks, and 37 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 36 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 276 more blocks parsl.executors.status_handling INFO: Scaling out by 276 blocks parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-17 04:03:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-17 04:03:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 04:03:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3600e8c07d15030eaeeba4b9b7aad61c with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/62fa2a40-d9b7-4de4-a1fb-70230e781331_isrForCrosstalkSources_1031700_36.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/62fa2a40-d9b7-4de4-a1fb-70230e781331_isrForCrosstalkSources_1031700_36.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 558031a7e7a4cbfdbaf654c4e6d8b24d with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b856bf29-a283-470d-84a4-68e0f19aa9d9_isrForCrosstalkSources_1031707_47.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b856bf29-a283-470d-84a4-68e0f19aa9d9_isrForCrosstalkSources_1031707_47.stderr 2024-03-17 04:03:47 proc_lsst.multi:170 [INFO] [multi] job_id 17147977 proc_lsst.multi INFO: [multi] job_id 17147977 2024-03-17 04:03:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:47 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147977 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147977 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17147977 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-17 04:03:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-17 04:03:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b31342e98114c963b6cdb670c34a3a0c with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/22ab7c7d-7251-4e56-8f3e-628e43be0fa2_isrForCrosstalkSources_1031713_59.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/22ab7c7d-7251-4e56-8f3e-628e43be0fa2_isrForCrosstalkSources_1031713_59.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 782e5af672c96d449a664c18eb7c8ab6 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/b45a8b6a-ac73-4c60-8b8a-bf5dcb745f59_isrForCrosstalkSources_1031729_49.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/b45a8b6a-ac73-4c60-8b8a-bf5dcb745f59_isrForCrosstalkSources_1031729_49.stderr 2024-03-17 04:03:48 proc_lsst.multi:170 [INFO] [multi] job_id 17147978 proc_lsst.multi INFO: [multi] job_id 17147978 2024-03-17 04:03:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 04:03:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:48 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147978 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147978 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17147978 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-17 04:03:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-17 04:03:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 04:03:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:03:49 proc_lsst.multi:170 [INFO] [multi] job_id 17147979 proc_lsst.multi INFO: [multi] job_id 17147979 2024-03-17 04:03:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 04:03:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:49 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147979 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147979 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17147979 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-17 04:03:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-17 04:03:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 04:03:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f123f00a9ffcfbd584c0a4c430d1b75a with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c7e7fba6-15bb-4149-a342-b885d2a7f811_isrForCrosstalkSources_1031725_51.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c7e7fba6-15bb-4149-a342-b885d2a7f811_isrForCrosstalkSources_1031725_51.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 676de10495b069eb5291ef6d4abce6c7 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/36cad2b1-3155-4eed-b8c0-149e55d08b07_isrForCrosstalkSources_1031727_17.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/36cad2b1-3155-4eed-b8c0-149e55d08b07_isrForCrosstalkSources_1031727_17.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dedcab6ccf2c49c1638a8c29177ff50 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0c7cf628-924a-4efe-80a7-e56a41b70a9e_isrForCrosstalkSources_1031721_4.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0c7cf628-924a-4efe-80a7-e56a41b70a9e_isrForCrosstalkSources_1031721_4.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3888f37c0db4f6b47faef126e4498c65 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/fea45413-0a38-4881-8dce-e8fd19813e40_isrForCrosstalkSources_1031733_52.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/fea45413-0a38-4881-8dce-e8fd19813e40_isrForCrosstalkSources_1031733_52.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry faf265ce8cb2ed9f13c1c3dc4f2abbcf with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/40d109fa-2d05-4118-b1dd-a4da6a9779d7_isrForCrosstalkSources_1031729_18.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/40d109fa-2d05-4118-b1dd-a4da6a9779d7_isrForCrosstalkSources_1031729_18.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90ba0ce2763e975cdc2f65a9b731bbef with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/ad1c3320-25ad-4e0c-abae-f73305f13216_isrForCrosstalkSources_1031701_48.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/ad1c3320-25ad-4e0c-abae-f73305f13216_isrForCrosstalkSources_1031701_48.stderr 2024-03-17 04:03:50 proc_lsst.multi:170 [INFO] [multi] job_id 17147980 proc_lsst.multi INFO: [multi] job_id 17147980 2024-03-17 04:03:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 04:03:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147980 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147980 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17147980 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-17 04:03:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-17 04:03:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 04:03:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b75c0aca0c68928bd41f5dc381efeedd with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/95f50193-d744-4e66-b3fb-a0bb541baf2d_isrForCrosstalkSources_1031721_11.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/95f50193-d744-4e66-b3fb-a0bb541baf2d_isrForCrosstalkSources_1031721_11.stderr 2024-03-17 04:03:51 proc_lsst.multi:170 [INFO] [multi] job_id 17147981 proc_lsst.multi INFO: [multi] job_id 17147981 2024-03-17 04:03:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147981 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147981 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17147981 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-17 04:03:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-17 04:03:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69f4bd0531d1a50c7d4f7d41d53c2d96 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/38cc467d-e139-4575-9c73-88b0ea2836e6_isrForCrosstalkSources_1031726_41.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/38cc467d-e139-4575-9c73-88b0ea2836e6_isrForCrosstalkSources_1031726_41.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ed465544de0d45d9d24a13f22246bdf with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/06fe5dfa-f6db-4597-961c-72ecb273d5db_isrForCrosstalkSources_1031725_55.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/06fe5dfa-f6db-4597-961c-72ecb273d5db_isrForCrosstalkSources_1031725_55.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c35c76993bd1d68ccc5246a69bbcef7f with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/763fb7ea-ed18-40e8-a164-6881a3557b41_isrForCrosstalkSources_1031707_1.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/763fb7ea-ed18-40e8-a164-6881a3557b41_isrForCrosstalkSources_1031707_1.stderr 2024-03-17 04:03:52 proc_lsst.multi:170 [INFO] [multi] job_id 17147983 proc_lsst.multi INFO: [multi] job_id 17147983 2024-03-17 04:03:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 04:03:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147983 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147983 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17147983 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-17 04:03:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-17 04:03:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 04:03:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75f940234f3ec3df12c7349eee62ce8d with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/cb32eddc-718b-488c-8f17-db9183c25e3c_isrForCrosstalkSources_1031718_52.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/cb32eddc-718b-488c-8f17-db9183c25e3c_isrForCrosstalkSources_1031718_52.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e906ff1b257d0d67986b4362320dac6 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c8239c0f-46c0-47e4-84fb-d07edf4f032d_isrForCrosstalkSources_1031719_41.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c8239c0f-46c0-47e4-84fb-d07edf4f032d_isrForCrosstalkSources_1031719_41.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47817507f3c8590597ee7ddea91e7b66 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/9f357df4-c8fb-4a8c-aef7-9e485eae0dfd_isrForCrosstalkSources_1031720_50.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/9f357df4-c8fb-4a8c-aef7-9e485eae0dfd_isrForCrosstalkSources_1031720_50.stderr 2024-03-17 04:03:54 proc_lsst.multi:170 [INFO] [multi] job_id 17147984 proc_lsst.multi INFO: [multi] job_id 17147984 2024-03-17 04:03:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 04:03:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:54 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147984 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147984 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17147984 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-17 04:03:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-17 04:03:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 04:03:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f413438fc156daabd17d4936eefaaa58 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/69509a94-a9b4-4dfa-b297-fd36b142d2d9_isrForCrosstalkSources_1031720_21.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/69509a94-a9b4-4dfa-b297-fd36b142d2d9_isrForCrosstalkSources_1031720_21.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94f560152b5da69239096d07c634e50b with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8d919904-c040-4c57-9d8f-dbae865cffeb_isrForCrosstalkSources_1031735_19.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8d919904-c040-4c57-9d8f-dbae865cffeb_isrForCrosstalkSources_1031735_19.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd1de0e66802aff46d5b8222f78a9430 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/45cddbe6-218c-4f98-bb2d-173d051225f3_isrForCrosstalkSources_1031720_38.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/45cddbe6-218c-4f98-bb2d-173d051225f3_isrForCrosstalkSources_1031720_38.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5baa63a8d7336d6e45f58946a20afdfd with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/90841de7-d469-4caf-bb34-56f6f024bd22_isrForCrosstalkSources_1031726_23.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/90841de7-d469-4caf-bb34-56f6f024bd22_isrForCrosstalkSources_1031726_23.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5e777dd02085281974625a3520ff140 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0493b1cc-0185-4915-9835-d2f1c04f9a16_isrForCrosstalkSources_1031719_38.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0493b1cc-0185-4915-9835-d2f1c04f9a16_isrForCrosstalkSources_1031719_38.stderr 2024-03-17 04:03:56 proc_lsst.multi:170 [INFO] [multi] job_id 17147985 proc_lsst.multi INFO: [multi] job_id 17147985 2024-03-17 04:03:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 04:03:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:56 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147985 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147985 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17147985 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-17 04:03:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-17 04:03:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 04:03:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d403ec491ea87723f293c58922bb5530 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d240881f-db7d-4fa9-be60-8ce92789a542_isrForCrosstalkSources_1031731_10.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d240881f-db7d-4fa9-be60-8ce92789a542_isrForCrosstalkSources_1031731_10.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b199a3e419ad93eb481e9d5d3f56255 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/70d36247-dad6-4f75-8b63-129ac42c86ce_isrForCrosstalkSources_1031702_60.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/70d36247-dad6-4f75-8b63-129ac42c86ce_isrForCrosstalkSources_1031702_60.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71b5900736f282e5c24d1b5b43562ac5 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks 2024-03-17 04:03:57 proc_lsst.multi:170 [INFO] [multi] job_id 17147986 proc_lsst.multi INFO: [multi] job_id 17147986 2024-03-17 04:03:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 04:03:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147986 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147986 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/632d85e8-72dd-4183-b922-4c8d9395b69e_isrForCrosstalkSources_1031711_27.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/632d85e8-72dd-4183-b922-4c8d9395b69e_isrForCrosstalkSources_1031711_27.stderr parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17147986 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-17 04:03:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-17 04:03:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 04:03:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25d5e143f6d2ed2a452deeed4f7997f5 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ed41865d-fbff-439f-a906-98bc807b45a0_isrForCrosstalkSources_1031715_51.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ed41865d-fbff-439f-a906-98bc807b45a0_isrForCrosstalkSources_1031715_51.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74d894e490e23f9a6a766c9ad88096cc with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f26ae15e-fe6a-4176-a370-56e9f6d0f6e9_isrForCrosstalkSources_1031720_40.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f26ae15e-fe6a-4176-a370-56e9f6d0f6e9_isrForCrosstalkSources_1031720_40.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5de65302ab5468141b3482f2cbd26074 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c724b541-fa99-4678-a1de-1a23da8f4edb_isrForCrosstalkSources_1031726_22.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c724b541-fa99-4678-a1de-1a23da8f4edb_isrForCrosstalkSources_1031726_22.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 127b430a21bb7ac64798a182a2d5afd4 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/84680fa7-8732-4e1a-9df7-7e84fc6e9202_isrForCrosstalkSources_1031738_23.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/84680fa7-8732-4e1a-9df7-7e84fc6e9202_isrForCrosstalkSources_1031738_23.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58807f3fb89895f36b47296112425642 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ff67b906-36e7-4f6e-97b6-cf9cc24b8743_isrForCrosstalkSources_1031729_55.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ff67b906-36e7-4f6e-97b6-cf9cc24b8743_isrForCrosstalkSources_1031729_55.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9d180f5afe2504b5418ae41f396b9c4 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/742e2c8d-be80-4438-af53-52e2ccd860ee_isrForCrosstalkSources_1031729_36.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/742e2c8d-be80-4438-af53-52e2ccd860ee_isrForCrosstalkSources_1031729_36.stderr 2024-03-17 04:03:59 proc_lsst.multi:170 [INFO] [multi] job_id 17147987 proc_lsst.multi INFO: [multi] job_id 17147987 2024-03-17 04:03:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 04:03:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147987 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147987 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17147987 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-17 04:03:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-17 04:03:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:03:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:03:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:03:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:03:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:03:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:03:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:03:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:03:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 04:03:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:03:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b90bf386272aa6bffcc757480706c7a with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3949d8af-61f0-4d8a-98ab-2849af551681_isrForCrosstalkSources_1031738_22.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3949d8af-61f0-4d8a-98ab-2849af551681_isrForCrosstalkSources_1031738_22.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82df5ed4242b9f9511588d7d5448ff67 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1598e231-3ce7-40e5-b5a5-b62d8eb29ba0_isrForCrosstalkSources_1031732_18.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1598e231-3ce7-40e5-b5a5-b62d8eb29ba0_isrForCrosstalkSources_1031732_18.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b42424197bb5b5ce9acf9f04b90acec with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/26f4215e-a5fe-4dbc-9261-4e0e68f3d4c6_isrForCrosstalkSources_1031711_41.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/26f4215e-a5fe-4dbc-9261-4e0e68f3d4c6_isrForCrosstalkSources_1031711_41.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aba818cf0f0085928c3254db53f50611 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/eb1608a1-98d6-446a-babb-ba83d46e9f4f_isrForCrosstalkSources_1031718_15.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/eb1608a1-98d6-446a-babb-ba83d46e9f4f_isrForCrosstalkSources_1031718_15.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6983761576a26f242b6a3faf8b02ecf with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/554b4c99-c73a-46e7-8958-188c8c1d0b9a_isrForCrosstalkSources_1031703_17.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/554b4c99-c73a-46e7-8958-188c8c1d0b9a_isrForCrosstalkSources_1031703_17.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04c32ea1db0d87b15babcf6a7c6d8ea1 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0b4ac318-fc9f-4bbc-a92f-0b5a581c2501_isrForCrosstalkSources_1031712_23.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0b4ac318-fc9f-4bbc-a92f-0b5a581c2501_isrForCrosstalkSources_1031712_23.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac254c8c6f25585e0cf518afef2e71fc with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/256e1196-44aa-4eff-90d2-a3ed8a18eae2_isrForCrosstalkSources_1031703_37.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/256e1196-44aa-4eff-90d2-a3ed8a18eae2_isrForCrosstalkSources_1031703_37.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 877ddcd28cc59f950400516547dc2892 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/585116bd-37be-41ee-bbf4-cbba6bca9355_isrForCrosstalkSources_1031717_5.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/585116bd-37be-41ee-bbf4-cbba6bca9355_isrForCrosstalkSources_1031717_5.stderr 2024-03-17 04:04:01 proc_lsst.multi:170 [INFO] [multi] job_id 17147988 proc_lsst.multi INFO: [multi] job_id 17147988 2024-03-17 04:04:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 04:04:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:01 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147988 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147988 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17147988 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-17 04:04:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-17 04:04:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 04:04:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b60aae93827d9c6cee4844bda01413c with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/4e4b4b4d-ca53-46eb-a0d7-b74132438b92_isrForCrosstalkSources_1031727_30.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/4e4b4b4d-ca53-46eb-a0d7-b74132438b92_isrForCrosstalkSources_1031727_30.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 960d8080fc05fa68370baf7617efabef with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/bcde156b-b66b-4b95-9f49-0dd11ec945d3_isrForCrosstalkSources_1031705_52.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/bcde156b-b66b-4b95-9f49-0dd11ec945d3_isrForCrosstalkSources_1031705_52.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ff646f2e00756377b7e58985ada297f with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/d2c0b056-fa64-4db6-aa28-22cf31fa54ef_isrForCrosstalkSources_1031724_31.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/d2c0b056-fa64-4db6-aa28-22cf31fa54ef_isrForCrosstalkSources_1031724_31.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eba3cd251cc021423566f0981baa762d with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4f4eb94e-ae95-4654-93f7-93f45824f02e_isrForCrosstalkSources_1031716_38.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4f4eb94e-ae95-4654-93f7-93f45824f02e_isrForCrosstalkSources_1031716_38.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210c805db24101936072794f01baaef2 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/1e56dedd-2a7c-4aad-93c1-3a55e5b72d78_isrForCrosstalkSources_1031738_46.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/1e56dedd-2a7c-4aad-93c1-3a55e5b72d78_isrForCrosstalkSources_1031738_46.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f40c13097e2d54636b52dac7197eab20 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/07c0b2ae-86d5-4390-90c5-4db883f5e670_isrForCrosstalkSources_1031729_56.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/07c0b2ae-86d5-4390-90c5-4db883f5e670_isrForCrosstalkSources_1031729_56.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f1456992ac402f7ec13ee63019d43ab with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3f9608cc-bc00-43b1-8a88-8c573b0dc553_isrForCrosstalkSources_1031707_49.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3f9608cc-bc00-43b1-8a88-8c573b0dc553_isrForCrosstalkSources_1031707_49.stderr 2024-03-17 04:04:03 proc_lsst.multi:170 [INFO] [multi] job_id 17147989 proc_lsst.multi INFO: [multi] job_id 17147989 2024-03-17 04:04:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-17 04:04:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147989 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147989 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17147989 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-17 04:04:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-17 04:04:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-17 04:04:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb876b4ac0e93da49d090227c96a3251 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/0b5f6a56-9426-4bdb-8c9f-3642dd1543fb_isrForCrosstalkSources_1031726_58.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/0b5f6a56-9426-4bdb-8c9f-3642dd1543fb_isrForCrosstalkSources_1031726_58.stderr 2024-03-17 04:04:05 proc_lsst.multi:170 [INFO] [multi] job_id 17147992 proc_lsst.multi INFO: [multi] job_id 17147992 2024-03-17 04:04:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-17 04:04:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:05 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147992 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147992 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17147992 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-17 04:04:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-17 04:04:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-17 04:04:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed6bc2fef0dfa3d15c2c58e7903fec2b with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2a90931b-0535-4973-8d66-767f30f07748_isrForCrosstalkSources_1031708_57.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2a90931b-0535-4973-8d66-767f30f07748_isrForCrosstalkSources_1031708_57.stderr 2024-03-17 04:04:06 proc_lsst.multi:170 [INFO] [multi] job_id 17147993 proc_lsst.multi INFO: [multi] job_id 17147993 2024-03-17 04:04:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-17 04:04:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:06 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147993 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147993 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17147993 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-17 04:04:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-17 04:04:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-17 04:04:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9d1b2f344cb10b838e306661f899145 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/bb6fc07b-fde2-4d1a-bfbe-8f258c001a06_isrForCrosstalkSources_1031708_20.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/bb6fc07b-fde2-4d1a-bfbe-8f258c001a06_isrForCrosstalkSources_1031708_20.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69204982ac1e27db4c27da6df8c1dac7 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/64306a60-cb9d-478b-9b46-c22facb3a691_isrForCrosstalkSources_1031706_58.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/64306a60-cb9d-478b-9b46-c22facb3a691_isrForCrosstalkSources_1031706_58.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c335d3634eeb13a1b32947a6962d110c with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/df4bdcd9-f35c-4336-b5b8-ccff721f2916_isrForCrosstalkSources_1031713_20.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/df4bdcd9-f35c-4336-b5b8-ccff721f2916_isrForCrosstalkSources_1031713_20.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 544cdae9ef2a9da59039f06239970015 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/4c0e8844-7641-4c35-adc0-920baa692150_isrForCrosstalkSources_1031734_34.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/4c0e8844-7641-4c35-adc0-920baa692150_isrForCrosstalkSources_1031734_34.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6552cdda8a1e40a91de6a74459d8d2b with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/a5942f87-2957-4fd9-b717-e6b39462a414_isrForCrosstalkSources_1031715_39.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/a5942f87-2957-4fd9-b717-e6b39462a414_isrForCrosstalkSources_1031715_39.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de6b7283b660346007c58d98a98ee967 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a75c7288-0928-42a7-b7bd-d9cc8b08adde_isrForCrosstalkSources_1031706_24.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a75c7288-0928-42a7-b7bd-d9cc8b08adde_isrForCrosstalkSources_1031706_24.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09872b6d013e556aeaf45406440cde7f with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f4df130b-1940-49b6-8a56-fd1fda09f559_isrForCrosstalkSources_1031727_12.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f4df130b-1940-49b6-8a56-fd1fda09f559_isrForCrosstalkSources_1031727_12.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2133946618dba3e4447afe243ef09cb7 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/17d7ff65-0ae6-4176-a1c5-4a3bce7e3743_isrForCrosstalkSources_1031705_11.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/17d7ff65-0ae6-4176-a1c5-4a3bce7e3743_isrForCrosstalkSources_1031705_11.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc923142657a183f6dfc7055a2d8d4b4 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0e4357f5-0154-4096-bd6d-166e81b0fe43_isrForCrosstalkSources_1031706_62.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0e4357f5-0154-4096-bd6d-166e81b0fe43_isrForCrosstalkSources_1031706_62.stderr 2024-03-17 04:04:08 proc_lsst.multi:170 [INFO] [multi] job_id 17147994 proc_lsst.multi INFO: [multi] job_id 17147994 2024-03-17 04:04:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-17 04:04:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:08 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147994 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147994 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17147994 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-17 04:04:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-17 04:04:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-17 04:04:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91db99a10e6d24d84201033316e37d60 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8ceb9b0b-06f5-4018-b679-d030f48a0227_isrForCrosstalkSources_1031704_43.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8ceb9b0b-06f5-4018-b679-d030f48a0227_isrForCrosstalkSources_1031704_43.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff13c1d809fd473530d295cd349f25f2 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/61cd904b-1275-432f-b592-44a3c831497b_isrForCrosstalkSources_1031714_10.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/61cd904b-1275-432f-b592-44a3c831497b_isrForCrosstalkSources_1031714_10.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42f2f9a243e8381737c4da454cc7e57b with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/91fcf096-8b1a-4ecb-89fa-75b35cbdccb8_isrForCrosstalkSources_1031738_8.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/91fcf096-8b1a-4ecb-89fa-75b35cbdccb8_isrForCrosstalkSources_1031738_8.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6535aa98a827adc0e50fd4b52e4d99f with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/600c5c00-68eb-4806-8b3b-f12a7a2d62ef_isrForCrosstalkSources_1031730_33.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/600c5c00-68eb-4806-8b3b-f12a7a2d62ef_isrForCrosstalkSources_1031730_33.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12ef5440976ea40d0b46e2b8fceb6582 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/f66bc53e-f9db-465d-b94b-a4ecd7f7afdd_isrForCrosstalkSources_1031719_48.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/f66bc53e-f9db-465d-b94b-a4ecd7f7afdd_isrForCrosstalkSources_1031719_48.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd042ed225a9969eb39eeb7ca0fc6fcc with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/20161884-7164-4c31-8bce-6ddc2d7461a8_isrForCrosstalkSources_1031708_46.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/20161884-7164-4c31-8bce-6ddc2d7461a8_isrForCrosstalkSources_1031708_46.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b59ca6a58cc056d9bd500107a5a6d0a with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/4fba5702-6669-4b33-9dd7-44c699b86990_isrForCrosstalkSources_1031714_34.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/4fba5702-6669-4b33-9dd7-44c699b86990_isrForCrosstalkSources_1031714_34.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe3f108e52890ce9943690d3d3aa51fa with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/4c1545db-8c72-454b-9110-3c911ea958f1_isrForCrosstalkSources_1031701_1.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/4c1545db-8c72-454b-9110-3c911ea958f1_isrForCrosstalkSources_1031701_1.stderr 2024-03-17 04:04:09 proc_lsst.multi:170 [INFO] [multi] job_id 17147995 proc_lsst.multi INFO: [multi] job_id 17147995 2024-03-17 04:04:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-17 04:04:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:09 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147995 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147995 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17147995 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-17 04:04:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-17 04:04:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-17 04:04:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13c977cb81bd4d045b13e2f0074107c0 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c0a79642-ad2e-4bb3-9598-83e75042c57e_isrForCrosstalkSources_1031738_17.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c0a79642-ad2e-4bb3-9598-83e75042c57e_isrForCrosstalkSources_1031738_17.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a73cecd12f5b277355775c046b305f4 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/0fbf0f12-18a0-4988-82d3-7d51ba8b96b8_isrForCrosstalkSources_1031701_41.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/0fbf0f12-18a0-4988-82d3-7d51ba8b96b8_isrForCrosstalkSources_1031701_41.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d0a2e0c247683ee90fbaa9cc50ebdcc with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/9041cd90-f65e-4454-8d91-c7621237a094_isrForCrosstalkSources_1031719_5.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/9041cd90-f65e-4454-8d91-c7621237a094_isrForCrosstalkSources_1031719_5.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05b511ca50f854e8e073a8107c8128ae with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/25039d19-68a7-472a-bfcc-824a93c469b0_isrForCrosstalkSources_1031724_28.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/25039d19-68a7-472a-bfcc-824a93c469b0_isrForCrosstalkSources_1031724_28.stderr 2024-03-17 04:04:11 proc_lsst.multi:170 [INFO] [multi] job_id 17147996 proc_lsst.multi INFO: [multi] job_id 17147996 2024-03-17 04:04:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-17 04:04:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147996 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147996 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17147996 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-17 04:04:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-17 04:04:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-17 04:04:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e715c2101b51263cedbc13c2318039d9 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bb7e27c5-9cfb-4d9c-8611-8c3b5fb05646_isrForCrosstalkSources_1031715_42.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bb7e27c5-9cfb-4d9c-8611-8c3b5fb05646_isrForCrosstalkSources_1031715_42.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7dc9fd5982980a138585c582ea38582 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1b1afef2-9423-4d2b-8619-2ade05af79f3_isrForCrosstalkSources_1031706_2.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1b1afef2-9423-4d2b-8619-2ade05af79f3_isrForCrosstalkSources_1031706_2.stderr 2024-03-17 04:04:13 proc_lsst.multi:170 [INFO] [multi] job_id 17147997 proc_lsst.multi INFO: [multi] job_id 17147997 2024-03-17 04:04:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-17 04:04:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147997 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147997 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17147997 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-17 04:04:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-17 04:04:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-17 04:04:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b72e3c50b37a5f520a174d6a506efd85 with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ab980b2c-16f4-4ed3-852f-987234150c3b_isrForCrosstalkSources_1031729_60.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/ab980b2c-16f4-4ed3-852f-987234150c3b_isrForCrosstalkSources_1031729_60.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c494dee00f6ccfed1bf2592c786386fd with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9c3bfeb4-07e1-4474-9424-b764f0781f86_isrForCrosstalkSources_1031718_41.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9c3bfeb4-07e1-4474-9424-b764f0781f86_isrForCrosstalkSources_1031718_41.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1edbafd903c699673df2fe78de904f56 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6032e216-305e-4866-8ce3-777dc515000b_isrForCrosstalkSources_1031723_36.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6032e216-305e-4866-8ce3-777dc515000b_isrForCrosstalkSources_1031723_36.stderr 2024-03-17 04:04:14 proc_lsst.multi:170 [INFO] [multi] job_id 17147998 proc_lsst.multi INFO: [multi] job_id 17147998 2024-03-17 04:04:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-17 04:04:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17147998 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17147998 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17147998 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-17 04:04:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-17 04:04:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-17 04:04:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3756f8e07b0dd11ef8368eaa63a45ca2 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/10b16b26-18c1-48e3-bdc8-987f07e2dbf9_isrForCrosstalkSources_1031714_27.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/10b16b26-18c1-48e3-bdc8-987f07e2dbf9_isrForCrosstalkSources_1031714_27.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb9aaa7997c8fb10a831e42943dc96ac with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/f69d4123-d46d-4946-90f4-b257795f18c4_isrForCrosstalkSources_1031725_34.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/f69d4123-d46d-4946-90f4-b257795f18c4_isrForCrosstalkSources_1031725_34.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e41d7a5e2eeb0cf55fd26b43ac936260 with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/697f3465-4a7c-4c8a-8fd5-406128457ebb_isrForCrosstalkSources_1031731_34.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/697f3465-4a7c-4c8a-8fd5-406128457ebb_isrForCrosstalkSources_1031731_34.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6eaa04ac3c227fb79c3d311a275a796 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/87885509-2836-4cb0-8d54-92ba336f3dcc_isrForCrosstalkSources_1031736_11.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/87885509-2836-4cb0-8d54-92ba336f3dcc_isrForCrosstalkSources_1031736_11.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2d422d79ada674c6bded4829f539000 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a4c4d095-c222-46a9-9193-592a858aedff_isrForCrosstalkSources_1031731_28.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a4c4d095-c222-46a9-9193-592a858aedff_isrForCrosstalkSources_1031731_28.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c36139506b694b0cbb5be4f2d64acda5 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0f934b63-1950-4fcf-b605-fabe54658492_isrForCrosstalkSources_1031720_46.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0f934b63-1950-4fcf-b605-fabe54658492_isrForCrosstalkSources_1031720_46.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dcde98f14404c1eff5f48405a7810f3 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3b35df55-e4ec-4fa7-a63d-219c3c8a4607_isrForCrosstalkSources_1031735_41.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3b35df55-e4ec-4fa7-a63d-219c3c8a4607_isrForCrosstalkSources_1031735_41.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e02f8cc812dd500d042cb7186108955a with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/05605a98-32aa-495d-b39c-1498b00e7894_isrForCrosstalkSources_1031735_12.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/05605a98-32aa-495d-b39c-1498b00e7894_isrForCrosstalkSources_1031735_12.stderr 2024-03-17 04:04:15 proc_lsst.multi:170 [INFO] [multi] job_id 17148000 proc_lsst.multi INFO: [multi] job_id 17148000 2024-03-17 04:04:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17148000 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17148000 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17148000 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-17 04:04:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-17 04:04:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 04:04:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9428c0ae1d36c5b6f31758203afb217 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/4399d1b4-2792-4a10-9494-df81ca5112df_isrForCrosstalkSources_1031734_54.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/4399d1b4-2792-4a10-9494-df81ca5112df_isrForCrosstalkSources_1031734_54.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8e268efcc50b5f3dcb938c8c32438ac with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d9c3f317-aaa9-44a3-a00b-b95d4f9b9e24_isrForCrosstalkSources_1031734_4.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d9c3f317-aaa9-44a3-a00b-b95d4f9b9e24_isrForCrosstalkSources_1031734_4.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d01f793b6caa4b155e02187eca9ad0dc with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7b69e4a0-11b7-4fe3-9491-5f791a1d4424_isrForCrosstalkSources_1031702_15.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7b69e4a0-11b7-4fe3-9491-5f791a1d4424_isrForCrosstalkSources_1031702_15.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6e6fa4396ee8e976a46b2f8c5833b2f with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3817f605-5510-483a-a68d-5f93548141ba_isrForCrosstalkSources_1031716_46.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3817f605-5510-483a-a68d-5f93548141ba_isrForCrosstalkSources_1031716_46.stderr 2024-03-17 04:04:16 proc_lsst.multi:170 [INFO] [multi] job_id 17148001 proc_lsst.multi INFO: [multi] job_id 17148001 2024-03-17 04:04:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148001 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148001 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17148001 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-17 04:04:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-17 04:04:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27c71b1da7f5929ebe1c53feddfd7d18 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/56f18e4e-1684-4d7e-ac17-fdce6ae3b795_isrForCrosstalkSources_1031729_52.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/56f18e4e-1684-4d7e-ac17-fdce6ae3b795_isrForCrosstalkSources_1031729_52.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fadee834033094b521b502a718cc916e with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a78ca919-e4c9-4324-bbdf-fa2956e5ea18_isrForCrosstalkSources_1031738_21.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a78ca919-e4c9-4324-bbdf-fa2956e5ea18_isrForCrosstalkSources_1031738_21.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d59ad2f9f40e91a603333fcd6315b16 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/b1b95903-8407-4aff-a0b9-2ffb0d1c2b72_isrForCrosstalkSources_1031699_46.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/b1b95903-8407-4aff-a0b9-2ffb0d1c2b72_isrForCrosstalkSources_1031699_46.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 539c09066a8bc65fb2dd52744a0b4997 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/338b2f46-a5a8-45a7-8dcd-6814b94a5616_isrForCrosstalkSources_1031715_28.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/338b2f46-a5a8-45a7-8dcd-6814b94a5616_isrForCrosstalkSources_1031715_28.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5128d60da76107af4d85e435181e7d97 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/e367b07b-e231-4aed-b32c-bdec41424c00_isrForCrosstalkSources_1031719_26.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/e367b07b-e231-4aed-b32c-bdec41424c00_isrForCrosstalkSources_1031719_26.stderr 2024-03-17 04:04:18 proc_lsst.multi:170 [INFO] [multi] job_id 17148002 proc_lsst.multi INFO: [multi] job_id 17148002 2024-03-17 04:04:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 04:04:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148002 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148002 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17148002 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-17 04:04:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-17 04:04:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 04:04:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da0d316b60b6110e78bb147f40612798 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7d591371-7e43-434c-bf0f-99f6240368e1_isrForCrosstalkSources_1031732_10.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7d591371-7e43-434c-bf0f-99f6240368e1_isrForCrosstalkSources_1031732_10.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91b7164e85780477ee015269f83e7076 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/cac6da4c-7f0a-477b-b018-1fc150b5274d_isrForCrosstalkSources_1031702_59.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/cac6da4c-7f0a-477b-b018-1fc150b5274d_isrForCrosstalkSources_1031702_59.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23f5fd21b095ca3aa7a6926d8a98727b with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c450a3e6-f729-4993-9c60-ea632f1cd6ca_isrForCrosstalkSources_1031705_10.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c450a3e6-f729-4993-9c60-ea632f1cd6ca_isrForCrosstalkSources_1031705_10.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b761654bc40eb0629629e6d5a89e62f6 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/86db2fba-551f-442c-abdd-195d2c57c210_isrForCrosstalkSources_1031733_56.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/86db2fba-551f-442c-abdd-195d2c57c210_isrForCrosstalkSources_1031733_56.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cca9646743e9855cc65dec77f9fef23b with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/255e43a8-0a21-41c5-a7b5-913a6939b54f_isrForCrosstalkSources_1031735_23.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/255e43a8-0a21-41c5-a7b5-913a6939b54f_isrForCrosstalkSources_1031735_23.stderr 2024-03-17 04:04:19 proc_lsst.multi:170 [INFO] [multi] job_id 17148003 proc_lsst.multi INFO: [multi] job_id 17148003 2024-03-17 04:04:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 04:04:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148003 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148003 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17148003 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-17 04:04:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-17 04:04:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 04:04:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f03dc3ec6dbef3c9b7f832d2c6d5b36 with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c054566e-3b95-4d12-9417-859b39b1ba73_isrForCrosstalkSources_1031726_30.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c054566e-3b95-4d12-9417-859b39b1ba73_isrForCrosstalkSources_1031726_30.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9137b5725ec6414f16e786cccc58fb9 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9afc5ac4-72ee-4f25-abe7-10e7c2590a65_isrForCrosstalkSources_1031729_45.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9afc5ac4-72ee-4f25-abe7-10e7c2590a65_isrForCrosstalkSources_1031729_45.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38023e1475c9c427ef6553a9999a159e with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/8e54480a-e077-4954-8b59-e2645080f8ff_isrForCrosstalkSources_1031725_10.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/8e54480a-e077-4954-8b59-e2645080f8ff_isrForCrosstalkSources_1031725_10.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d077f02271571e405e4a7e2706ee1c2 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/ba839e26-5a9d-46ac-aaaa-7166fde00e22_isrForCrosstalkSources_1031736_34.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/ba839e26-5a9d-46ac-aaaa-7166fde00e22_isrForCrosstalkSources_1031736_34.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5e5b950c8c1c52d0a8c03d6fb0085a8 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/dfe0ae5b-d690-41aa-be86-ab8cdd2fd5f7_isrForCrosstalkSources_1031729_30.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/dfe0ae5b-d690-41aa-be86-ab8cdd2fd5f7_isrForCrosstalkSources_1031729_30.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc4ccdb8a5bcbab429431a4b4b5fe2c5 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b7742569-a3ea-4422-8711-747fc79ba280_isrForCrosstalkSources_1031726_8.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b7742569-a3ea-4422-8711-747fc79ba280_isrForCrosstalkSources_1031726_8.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67b3f5dfa5cb19b031e856329e91a969 with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/decc39da-8a5a-4dcd-9999-d42fc74af517_isrForCrosstalkSources_1031718_18.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/decc39da-8a5a-4dcd-9999-d42fc74af517_isrForCrosstalkSources_1031718_18.stderr 2024-03-17 04:04:21 proc_lsst.multi:170 [INFO] [multi] job_id 17148004 proc_lsst.multi INFO: [multi] job_id 17148004 2024-03-17 04:04:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 04:04:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148004 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148004 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17148004 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-17 04:04:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-17 04:04:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 04:04:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a22aae3d73e5f3515dc9e8ad57daec85 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/bffb086e-4664-410c-ad48-a31a83b8baa5_isrForCrosstalkSources_1031706_49.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/bffb086e-4664-410c-ad48-a31a83b8baa5_isrForCrosstalkSources_1031706_49.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 702bbd3c48fb89725b411500ec6e6572 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d234ae18-87a8-4473-9727-7240336e9ffb_isrForCrosstalkSources_1031704_13.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/d234ae18-87a8-4473-9727-7240336e9ffb_isrForCrosstalkSources_1031704_13.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07a1ee57c1a20840d29897912014c242 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4a9f3117-7fe3-4cf1-b83a-142b0d81fb9b_isrForCrosstalkSources_1031699_54.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4a9f3117-7fe3-4cf1-b83a-142b0d81fb9b_isrForCrosstalkSources_1031699_54.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8401cb08a4e42b8f50192a30f7a14752 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b97ee244-a90b-4ef8-9e13-115434848bba_isrForCrosstalkSources_1031726_48.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b97ee244-a90b-4ef8-9e13-115434848bba_isrForCrosstalkSources_1031726_48.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e1f6f6d710a5a844d995ddc455b1658 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/39312ad2-42b0-45a2-98c3-420e7494f672_isrForCrosstalkSources_1031720_17.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/39312ad2-42b0-45a2-98c3-420e7494f672_isrForCrosstalkSources_1031720_17.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b1eea3a21a2dd90ca6a0b48365b6fdf with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/01c4e492-0f4a-4d6c-9449-e25d8718b868_isrForCrosstalkSources_1031738_27.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/01c4e492-0f4a-4d6c-9449-e25d8718b868_isrForCrosstalkSources_1031738_27.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49d3c88a9b5093e1f572e78b73aa91b3 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ca536097-50ae-4545-b89d-713e1ed3f966_isrForCrosstalkSources_1031713_29.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ca536097-50ae-4545-b89d-713e1ed3f966_isrForCrosstalkSources_1031713_29.stderr 2024-03-17 04:04:23 proc_lsst.multi:170 [INFO] [multi] job_id 17148005 proc_lsst.multi INFO: [multi] job_id 17148005 2024-03-17 04:04:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148005 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148005 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17148005 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-17 04:04:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-17 04:04:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92351f89f86c33471741014202ef9494 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67116442-5240-4ac0-8b23-99d1a0502d8b_isrForCrosstalkSources_1031728_29.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67116442-5240-4ac0-8b23-99d1a0502d8b_isrForCrosstalkSources_1031728_29.stderr 2024-03-17 04:04:24 proc_lsst.multi:170 [INFO] [multi] job_id 17148006 proc_lsst.multi INFO: [multi] job_id 17148006 2024-03-17 04:04:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 04:04:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148006 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148006 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17148006 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-17 04:04:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-17 04:04:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 04:04:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bac0e84df06be105f9ff3b35a60dbbe1 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/369b8bd8-2e7f-4a99-9830-7d13dc606efa_isrForCrosstalkSources_1031721_26.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/369b8bd8-2e7f-4a99-9830-7d13dc606efa_isrForCrosstalkSources_1031721_26.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a3cb37f32ab3634a825a02cc984ee03 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2e4140a2-9980-4f0d-9c9f-3b834d92cba2_isrForCrosstalkSources_1031709_59.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2e4140a2-9980-4f0d-9c9f-3b834d92cba2_isrForCrosstalkSources_1031709_59.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 752d2fe2fa0be2a35b371972873eded4 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1139df6e-0d11-46b6-88a6-ee62ba724dae_isrForCrosstalkSources_1031713_62.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1139df6e-0d11-46b6-88a6-ee62ba724dae_isrForCrosstalkSources_1031713_62.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c1948c8a9d5525f35e5a0c4eda7db3a with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f19537d0-12fe-402f-abef-b34aa317021c_isrForCrosstalkSources_1031735_1.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f19537d0-12fe-402f-abef-b34aa317021c_isrForCrosstalkSources_1031735_1.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0487716917a1959298109f2b7c8c378 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/eb05d2d2-9a7e-4c06-b4ab-c8700ce583a8_isrForCrosstalkSources_1031705_37.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/eb05d2d2-9a7e-4c06-b4ab-c8700ce583a8_isrForCrosstalkSources_1031705_37.stderr 2024-03-17 04:04:26 proc_lsst.multi:170 [INFO] [multi] job_id 17148007 proc_lsst.multi INFO: [multi] job_id 17148007 2024-03-17 04:04:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 04:04:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148007 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148007 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17148007 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-17 04:04:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-17 04:04:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 04:04:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bb09de0dd7bb1bee58da75cab4765f5 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/33fddf2c-d184-42bc-9f0f-58b3753fcdd2_isrForCrosstalkSources_1031707_53.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/33fddf2c-d184-42bc-9f0f-58b3753fcdd2_isrForCrosstalkSources_1031707_53.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b9387d9ee5b301b908a420e9caa38b8 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/294a3296-b8d3-4386-b651-ce31ce5bc78e_isrForCrosstalkSources_1031710_6.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/294a3296-b8d3-4386-b651-ce31ce5bc78e_isrForCrosstalkSources_1031710_6.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a857bc6ca44b3ca5c58127b29782560f with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bc103cfb-1a2f-41db-ae7b-e2f2469f348d_isrForCrosstalkSources_1031734_29.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bc103cfb-1a2f-41db-ae7b-e2f2469f348d_isrForCrosstalkSources_1031734_29.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb320cb9ea535deb719f07f6fde0158b with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1050fc2d-1497-457e-aca5-6c298e0ff7f9_isrForCrosstalkSources_1031735_4.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1050fc2d-1497-457e-aca5-6c298e0ff7f9_isrForCrosstalkSources_1031735_4.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27453974be202a17d13c9ecb28f2384b with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d9de3c61-5f5a-4894-88ba-59190823befe_isrForCrosstalkSources_1031734_5.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d9de3c61-5f5a-4894-88ba-59190823befe_isrForCrosstalkSources_1031734_5.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c961c4a8442ffd5674825984442a97c6 with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/73abf7de-1a7f-4995-a802-62d9f47176fc_isrForCrosstalkSources_1031709_44.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/73abf7de-1a7f-4995-a802-62d9f47176fc_isrForCrosstalkSources_1031709_44.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb431f49c0ce1e3d452cb8b30cab55b7 with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f10ba1c5-9892-4dc1-9c58-5dfd1978956f_isrForCrosstalkSources_1031700_60.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f10ba1c5-9892-4dc1-9c58-5dfd1978956f_isrForCrosstalkSources_1031700_60.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58733ac8a8bb445fdf92ad1837156a2a with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/2aaca8b1-7f94-4f58-8ac6-5c5aee97a91e_isrForCrosstalkSources_1031706_3.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/2aaca8b1-7f94-4f58-8ac6-5c5aee97a91e_isrForCrosstalkSources_1031706_3.stderr 2024-03-17 04:04:27 proc_lsst.multi:170 [INFO] [multi] job_id 17148008 proc_lsst.multi INFO: [multi] job_id 17148008 2024-03-17 04:04:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 04:04:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148008 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148008 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17148008 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-17 04:04:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-17 04:04:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 04:04:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fb110f0c2d918e457e67bd1b2e99f9a with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/d89b32e8-ae99-43c1-aec9-7ca34eef05b8_isrForCrosstalkSources_1031723_22.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/d89b32e8-ae99-43c1-aec9-7ca34eef05b8_isrForCrosstalkSources_1031723_22.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77adb2825086c086d7579eabdf9fa2c5 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5ef739d0-5422-4a8c-8490-57b0319fcf99_isrForCrosstalkSources_1031727_14.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5ef739d0-5422-4a8c-8490-57b0319fcf99_isrForCrosstalkSources_1031727_14.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb39c39fefcef04d40aec3d1259c8f38 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f5685629-b750-43d8-af11-34ca8b26c4fa_isrForCrosstalkSources_1031732_25.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f5685629-b750-43d8-af11-34ca8b26c4fa_isrForCrosstalkSources_1031732_25.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df7fe2242bfaf2d08807e3ed1d303848 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/910d7746-f200-416a-a5aa-469ac87f3865_isrForCrosstalkSources_1031731_46.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/910d7746-f200-416a-a5aa-469ac87f3865_isrForCrosstalkSources_1031731_46.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f37773bf3eba710c4769f6be90f1378 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/0c333068-0a7a-4bdd-9c55-85d531d804f1_isrForCrosstalkSources_1031713_23.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/0c333068-0a7a-4bdd-9c55-85d531d804f1_isrForCrosstalkSources_1031713_23.stderr 2024-03-17 04:04:28 proc_lsst.multi:170 [INFO] [multi] job_id 17148009 proc_lsst.multi INFO: [multi] job_id 17148009 2024-03-17 04:04:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 04:04:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148009 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148009 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17148009 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-17 04:04:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-17 04:04:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 04:04:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19459dde110fe6bcc82791a123b1e7bc with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/1b1fd11a-8783-451b-86a7-7a0b2eec9b57_isrForCrosstalkSources_1031721_8.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/1b1fd11a-8783-451b-86a7-7a0b2eec9b57_isrForCrosstalkSources_1031721_8.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a70431c1e10a260d01417a223abe60b with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/2831b89f-265b-434a-80ed-0251154ac974_isrForCrosstalkSources_1031700_53.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/2831b89f-265b-434a-80ed-0251154ac974_isrForCrosstalkSources_1031700_53.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 342d46511197c83ff66653336ba7e10a with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9f0f1a8b-33e2-4f7b-824e-ae991801080a_isrForCrosstalkSources_1031716_10.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9f0f1a8b-33e2-4f7b-824e-ae991801080a_isrForCrosstalkSources_1031716_10.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef27b3b97039a8f554214dff92132b35 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/2516cc82-5c08-44d4-bddd-922f3e153dac_isrForCrosstalkSources_1031728_9.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/2516cc82-5c08-44d4-bddd-922f3e153dac_isrForCrosstalkSources_1031728_9.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7470cab877ddbd94e2c3263a4743209e with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/fff4e98d-1a6d-41fa-85ed-5901c678b436_isrForCrosstalkSources_1031704_40.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/fff4e98d-1a6d-41fa-85ed-5901c678b436_isrForCrosstalkSources_1031704_40.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc222cab1ebcfaa2fdee0769d453cacb with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2bb075b3-cede-4a14-9203-b4040d88d390_isrForCrosstalkSources_1031704_21.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2bb075b3-cede-4a14-9203-b4040d88d390_isrForCrosstalkSources_1031704_21.stderr 2024-03-17 04:04:30 proc_lsst.multi:170 [INFO] [multi] job_id 17148010 proc_lsst.multi INFO: [multi] job_id 17148010 2024-03-17 04:04:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 04:04:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148010 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148010 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17148010 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-17 04:04:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-17 04:04:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 04:04:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24723e3b547b25231cf686b1167fc154 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8653db51-d6b7-4921-a419-333ae67f3ac4_isrForCrosstalkSources_1031733_15.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8653db51-d6b7-4921-a419-333ae67f3ac4_isrForCrosstalkSources_1031733_15.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d06dae6338ffe88557bdf5687a7560d with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/6c451e91-8877-46f9-b099-a6cd315e8c05_isrForCrosstalkSources_1031701_9.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/6c451e91-8877-46f9-b099-a6cd315e8c05_isrForCrosstalkSources_1031701_9.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17551e697c9459c497066ca5a2ebab57 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3c653a90-9c94-44fd-825c-ea3ec87e1c66_isrForCrosstalkSources_1031731_25.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3c653a90-9c94-44fd-825c-ea3ec87e1c66_isrForCrosstalkSources_1031731_25.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b0a43f767066d9605da5f3b110d110d with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/f1e08434-d101-4699-8b51-64ea93490f0a_isrForCrosstalkSources_1031707_59.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/f1e08434-d101-4699-8b51-64ea93490f0a_isrForCrosstalkSources_1031707_59.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3c39fba4e636e3adeccfa6d80841f90 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/e5e9cecd-3e34-4df6-8991-46a7dbbbb459_isrForCrosstalkSources_1031724_8.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/e5e9cecd-3e34-4df6-8991-46a7dbbbb459_isrForCrosstalkSources_1031724_8.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a55c279b23b71ca846230079d6dd524f with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/b78cda99-9249-4b6c-a3a7-a4d41aa5e8da_isrForCrosstalkSources_1031731_4.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/b78cda99-9249-4b6c-a3a7-a4d41aa5e8da_isrForCrosstalkSources_1031731_4.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 109e515ac6ce98cd8bcf4be2750bcc17 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/84315f07-e141-40d9-a605-42692dbc3d76_isrForCrosstalkSources_1031699_26.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/84315f07-e141-40d9-a605-42692dbc3d76_isrForCrosstalkSources_1031699_26.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fd77320f249818ef498ebcbf476303c with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/5ff06e0f-b27e-4ac4-bc20-bd2ccc90ce2c_isrForCrosstalkSources_1031719_4.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/5ff06e0f-b27e-4ac4-bc20-bd2ccc90ce2c_isrForCrosstalkSources_1031719_4.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2e8c8369feb711d33128e2d8eeed162 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/b0a3bbde-75de-4db5-a62d-f207ccefb614_isrForCrosstalkSources_1031711_38.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/b0a3bbde-75de-4db5-a62d-f207ccefb614_isrForCrosstalkSources_1031711_38.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81256a402016d3d9582e8554782b1718 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d81bb3a3-b19a-4964-86d2-396bc1ebd907_isrForCrosstalkSources_1031728_34.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d81bb3a3-b19a-4964-86d2-396bc1ebd907_isrForCrosstalkSources_1031728_34.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f8a0e4b310b1c203de801ca45ddc5eb with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c238c19d-0a6b-4f9b-a774-e389d31e072c_isrForCrosstalkSources_1031726_38.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c238c19d-0a6b-4f9b-a774-e389d31e072c_isrForCrosstalkSources_1031726_38.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7472244cd9b12d6756c762f2f88ac701 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/436a93d5-36ca-4598-aefb-085bef7f37a3_isrForCrosstalkSources_1031704_34.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/436a93d5-36ca-4598-aefb-085bef7f37a3_isrForCrosstalkSources_1031704_34.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 392715047b8a45a3088c71f18135ff79 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/45462955-3be3-4241-bce6-423110fad208_isrForCrosstalkSources_1031715_52.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/45462955-3be3-4241-bce6-423110fad208_isrForCrosstalkSources_1031715_52.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07db0967d667bac801aa6f976371ba14 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/0d26086f-0400-4357-a8ce-958e5445893b_isrForCrosstalkSources_1031709_60.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/0d26086f-0400-4357-a8ce-958e5445893b_isrForCrosstalkSources_1031709_60.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4609188318f72466d0ec262201c92ab with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ec9ff92c-c3bf-4465-9811-a8a5ef228117_isrForCrosstalkSources_1031715_27.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ec9ff92c-c3bf-4465-9811-a8a5ef228117_isrForCrosstalkSources_1031715_27.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 997d264cb6af3c5175c38db47177af26 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/88f6663c-1461-4f83-a07b-580eff8e4798_isrForCrosstalkSources_1031712_61.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/88f6663c-1461-4f83-a07b-580eff8e4798_isrForCrosstalkSources_1031712_61.stderr 2024-03-17 04:04:33 proc_lsst.multi:170 [INFO] [multi] job_id 17148011 proc_lsst.multi INFO: [multi] job_id 17148011 2024-03-17 04:04:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 04:04:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148011 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148011 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17148011 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-17 04:04:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-17 04:04:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 04:04:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d281fa04c0501caa41c2717c6a232e2 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/36eae125-e68f-43d6-8809-619f7fcd45c7_isrForCrosstalkSources_1031707_60.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/36eae125-e68f-43d6-8809-619f7fcd45c7_isrForCrosstalkSources_1031707_60.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a303340e641d937d1b5643ced26c875e with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/8e4cf9fa-aa36-4cd8-82a5-da99eee67d1b_isrForCrosstalkSources_1031734_61.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/8e4cf9fa-aa36-4cd8-82a5-da99eee67d1b_isrForCrosstalkSources_1031734_61.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 987b5fedfa44926de012dda6a145a0da with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5a41a18f-24ec-49f4-bd32-ca8de2c5b490_isrForCrosstalkSources_1031717_37.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5a41a18f-24ec-49f4-bd32-ca8de2c5b490_isrForCrosstalkSources_1031717_37.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea6e855047cc364e35ce1cef62bd21cc with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0187ac5d-050e-45ea-a6f8-2b425fe3b4fb_isrForCrosstalkSources_1031705_2.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0187ac5d-050e-45ea-a6f8-2b425fe3b4fb_isrForCrosstalkSources_1031705_2.stderr 2024-03-17 04:04:35 proc_lsst.multi:170 [INFO] [multi] job_id 17148012 proc_lsst.multi INFO: [multi] job_id 17148012 2024-03-17 04:04:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-17 04:04:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148012 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148012 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17148012 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-17 04:04:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-17 04:04:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-17 04:04:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e1d65ade7f868480b86df3714335808 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/38186904-e514-46ca-ad7d-d04c6d626d99_isrForCrosstalkSources_1031724_50.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/38186904-e514-46ca-ad7d-d04c6d626d99_isrForCrosstalkSources_1031724_50.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53d7ed31aeded14b8b73ee54bcdd7c23 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/21c7f0c1-71ce-467d-bbab-5ffd69953038_isrForCrosstalkSources_1031723_53.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/21c7f0c1-71ce-467d-bbab-5ffd69953038_isrForCrosstalkSources_1031723_53.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a133386affa277c45f3199026a3b49b with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f07ae3c1-6343-4c11-9909-d6aa9922636a_isrForCrosstalkSources_1031716_58.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f07ae3c1-6343-4c11-9909-d6aa9922636a_isrForCrosstalkSources_1031716_58.stderr 2024-03-17 04:04:36 proc_lsst.multi:170 [INFO] [multi] job_id 17148013 proc_lsst.multi INFO: [multi] job_id 17148013 2024-03-17 04:04:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-17 04:04:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148013 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148013 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17148013 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-17 04:04:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-17 04:04:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-17 04:04:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 880031ecc7d9ed8598f287066b89a590 with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/357c3619-7bd7-43f4-a298-52bc04bde48c_isrForCrosstalkSources_1031699_62.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/357c3619-7bd7-43f4-a298-52bc04bde48c_isrForCrosstalkSources_1031699_62.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a603201de2f2322874afa8d4ba67796e with result from task 358 2024-03-17 04:04:37 proc_lsst.multi:170 [INFO] [multi] job_id 17148014 proc_lsst.multi INFO: [multi] job_id 17148014 2024-03-17 04:04:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-17 04:04:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148014 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148014 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17148014 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-17 04:04:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-17 04:04:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-17 04:04:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a746de3c-11b4-4e95-8e8f-91056c854f95_isrForCrosstalkSources_1031713_49.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a746de3c-11b4-4e95-8e8f-91056c854f95_isrForCrosstalkSources_1031713_49.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c7d8baec9519af9f399dbbef2cb079e with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a8812683-2f24-4e9a-b6e0-9dbb86ea4f69_isrForCrosstalkSources_1031720_31.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a8812683-2f24-4e9a-b6e0-9dbb86ea4f69_isrForCrosstalkSources_1031720_31.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b623d9aaa6328108bb5b32594cabfd6 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5515bee1-eb47-42b1-816e-58fe233eeeb8_isrForCrosstalkSources_1031717_33.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5515bee1-eb47-42b1-816e-58fe233eeeb8_isrForCrosstalkSources_1031717_33.stderr parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cec234855416715113c64da010f7fc2 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9525813a-6289-458a-baed-72106b4af81b_isrForCrosstalkSources_1031730_32.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9525813a-6289-458a-baed-72106b4af81b_isrForCrosstalkSources_1031730_32.stderr parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9e74d63651d0ca02a69e512c9be255d with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/14891035-47c8-4868-a7d9-5deb83468188_isrForCrosstalkSources_1031727_51.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/14891035-47c8-4868-a7d9-5deb83468188_isrForCrosstalkSources_1031727_51.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9fbc0cb28f7319b790f3063f4a3a5a9 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d9de4f7d-399c-4df5-8586-eab7af19b484_isrForCrosstalkSources_1031738_40.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d9de4f7d-399c-4df5-8586-eab7af19b484_isrForCrosstalkSources_1031738_40.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b69b7b60f208b36838453600f0807eb8 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/b356972c-d1c5-4915-b6a1-3e18016a7bad_isrForCrosstalkSources_1031732_55.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/b356972c-d1c5-4915-b6a1-3e18016a7bad_isrForCrosstalkSources_1031732_55.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bb73a41e937637be8f1ebe6eeccced2 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/34b0110c-c0e4-47a3-927e-03c39b0c674f_isrForCrosstalkSources_1031723_21.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/34b0110c-c0e4-47a3-927e-03c39b0c674f_isrForCrosstalkSources_1031723_21.stderr 2024-03-17 04:04:38 proc_lsst.multi:170 [INFO] [multi] job_id 17148015 proc_lsst.multi INFO: [multi] job_id 17148015 2024-03-17 04:04:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-17 04:04:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148015 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148015 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17148015 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-17 04:04:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-17 04:04:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83cb2df445981e8859032313adbd5f6a with result from task 377 proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-17 04:04:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0e5c80d7-7466-4c2f-9be5-ae895c080328_isrForCrosstalkSources_1031721_13.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0e5c80d7-7466-4c2f-9be5-ae895c080328_isrForCrosstalkSources_1031721_13.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e253a6c8e9ee62ec3fadc30a251c16 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/acb4261d-e9f9-48bb-a925-da5d2d76de97_isrForCrosstalkSources_1031718_21.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/acb4261d-e9f9-48bb-a925-da5d2d76de97_isrForCrosstalkSources_1031718_21.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cd3b6312bc9fe75bc0e97d6586b7f0d with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1263c925-8207-47c0-a52a-970f30b5668a_isrForCrosstalkSources_1031711_23.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1263c925-8207-47c0-a52a-970f30b5668a_isrForCrosstalkSources_1031711_23.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9b1992d884309b5ce567462c2bb2d4a with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a14f61d9-a7a6-475f-852b-e7783130497e_isrForCrosstalkSources_1031711_21.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a14f61d9-a7a6-475f-852b-e7783130497e_isrForCrosstalkSources_1031711_21.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8549f0ab7d4f97347273b1c70a806bbb with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/45db14f5-f5c1-4df3-be92-937def6e74cd_isrForCrosstalkSources_1031704_4.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/45db14f5-f5c1-4df3-be92-937def6e74cd_isrForCrosstalkSources_1031704_4.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc2175e071f7558c0f61c3c80f374436 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/669191a9-b24f-42c3-ae0b-58c567548e28_isrForCrosstalkSources_1031714_35.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/669191a9-b24f-42c3-ae0b-58c567548e28_isrForCrosstalkSources_1031714_35.stderr 2024-03-17 04:04:39 proc_lsst.multi:170 [INFO] [multi] job_id 17148016 proc_lsst.multi INFO: [multi] job_id 17148016 2024-03-17 04:04:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-17 04:04:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148016 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148016 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17148016 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-17 04:04:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-17 04:04:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-17 04:04:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3836f24ccfad6348ee3c35cde2883a92 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d9b8ddfc-d4a6-4ea4-8e57-f2722409af1f_isrForCrosstalkSources_1031714_42.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d9b8ddfc-d4a6-4ea4-8e57-f2722409af1f_isrForCrosstalkSources_1031714_42.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf0968078bd561f2574c2e40e2c9ec6 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d6a125d4-686e-4ab8-af20-183d1900ceb5_isrForCrosstalkSources_1031734_1.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d6a125d4-686e-4ab8-af20-183d1900ceb5_isrForCrosstalkSources_1031734_1.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be2859e939168450503bb8fbf685d437 with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8693b0c3-b364-44b3-a0dc-b44efc9bbef2_isrForCrosstalkSources_1031700_6.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8693b0c3-b364-44b3-a0dc-b44efc9bbef2_isrForCrosstalkSources_1031700_6.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7b1e09ebfd0e12aa0215c4fba3bfe0e with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f54ba8a4-6b88-4925-938f-7a4b35fbfb70_isrForCrosstalkSources_1031709_27.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f54ba8a4-6b88-4925-938f-7a4b35fbfb70_isrForCrosstalkSources_1031709_27.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d48b4d4f97d751a6546e3d07e8172ef with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/5e2cce73-b6ca-4fe5-9665-ffde49f4a57d_isrForCrosstalkSources_1031718_38.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/5e2cce73-b6ca-4fe5-9665-ffde49f4a57d_isrForCrosstalkSources_1031718_38.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63ab57ee47f05cc8e4db6c697eaca768 with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c555a57f-0090-4a43-8091-a4333390327e_isrForCrosstalkSources_1031718_28.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c555a57f-0090-4a43-8091-a4333390327e_isrForCrosstalkSources_1031718_28.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e5adf47f549474437a2a25298a68c76 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/65aa0be0-2f0f-4e6b-b432-1392b049714a_isrForCrosstalkSources_1031732_1.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/65aa0be0-2f0f-4e6b-b432-1392b049714a_isrForCrosstalkSources_1031732_1.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bce542245e0b9527ebbc5304679b0c53 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/1b2a5402-b27d-48f6-af04-4a7cb45afd48_isrForCrosstalkSources_1031707_34.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/1b2a5402-b27d-48f6-af04-4a7cb45afd48_isrForCrosstalkSources_1031707_34.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07635ac62bcdf9bda04f025c538400e8 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/18d55a88-b62d-4329-b315-6d15c17eab7c_isrForCrosstalkSources_1031705_36.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/18d55a88-b62d-4329-b315-6d15c17eab7c_isrForCrosstalkSources_1031705_36.stderr 2024-03-17 04:04:41 proc_lsst.multi:170 [INFO] [multi] job_id 17148017 proc_lsst.multi INFO: [multi] job_id 17148017 2024-03-17 04:04:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-17 04:04:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148017 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148017 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17148017 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-17 04:04:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-17 04:04:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-17 04:04:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7363317be69bb9eccee6a6a3ae9e6d7a with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/9d8f524c-596d-4c8a-a905-44a07ec7bffa_isrForCrosstalkSources_1031736_2.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/9d8f524c-596d-4c8a-a905-44a07ec7bffa_isrForCrosstalkSources_1031736_2.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51d74483e875158b6f5ab991abfaecb6 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/29d4639e-f90f-424d-a7ee-35e4767fd29c_isrForCrosstalkSources_1031708_45.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/29d4639e-f90f-424d-a7ee-35e4767fd29c_isrForCrosstalkSources_1031708_45.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7711b2205c7c2e1ad023571863014fc4 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/c1ed0e56-0706-45c1-8e0b-d4f4173a368b_isrForCrosstalkSources_1031736_44.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/c1ed0e56-0706-45c1-8e0b-d4f4173a368b_isrForCrosstalkSources_1031736_44.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c6b12bea92e73c5f19e0e06c4784d3a with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1e57ec1f-b209-45e5-a986-739aa777354e_isrForCrosstalkSources_1031706_28.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1e57ec1f-b209-45e5-a986-739aa777354e_isrForCrosstalkSources_1031706_28.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5efe29d0e0ece89d7736f0eacc635374 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/e7f916df-fa99-43b6-aefb-77879d13268d_isrForCrosstalkSources_1031718_53.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/e7f916df-fa99-43b6-aefb-77879d13268d_isrForCrosstalkSources_1031718_53.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b968fab45018ffa1fe1123c06d0a2c9c with result from task 389 2024-03-17 04:04:43 proc_lsst.multi:170 [INFO] [multi] job_id 17148018 proc_lsst.multi INFO: [multi] job_id 17148018 2024-03-17 04:04:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-17 04:04:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148018 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148018 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17148018 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-17 04:04:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-17 04:04:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3d5ccbf6-f180-490a-9848-6ef1b8ca5c74_isrForCrosstalkSources_1031736_36.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3d5ccbf6-f180-490a-9848-6ef1b8ca5c74_isrForCrosstalkSources_1031736_36.stderr proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-17 04:04:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c3139f0ff29a9a04d0bb380c2862e5e with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/128feb0c-c466-44e7-8d5b-79015596dd35_isrForCrosstalkSources_1031728_42.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/128feb0c-c466-44e7-8d5b-79015596dd35_isrForCrosstalkSources_1031728_42.stderr parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0702ae8f3d2e98d3dfc929858fb70739 with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a61509b5-b0b5-49d7-a20b-620e2d867601_isrForCrosstalkSources_1031723_14.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a61509b5-b0b5-49d7-a20b-620e2d867601_isrForCrosstalkSources_1031723_14.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a701f7932ba07fcf5f8d622bd7aec60 with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c2aa801b-7583-4e01-9538-d2e9be66747c_isrForCrosstalkSources_1031718_55.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c2aa801b-7583-4e01-9538-d2e9be66747c_isrForCrosstalkSources_1031718_55.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee4df680408b3448d826cfcc58c22402 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a6cf886d-b598-4d3a-8874-019bb9e57a95_isrForCrosstalkSources_1031722_47.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a6cf886d-b598-4d3a-8874-019bb9e57a95_isrForCrosstalkSources_1031722_47.stderr 2024-03-17 04:04:44 proc_lsst.multi:170 [INFO] [multi] job_id 17148019 proc_lsst.multi INFO: [multi] job_id 17148019 2024-03-17 04:04:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-17 04:04:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148019 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148019 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17148019 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-17 04:04:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-17 04:04:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-17 04:04:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a0fd4798e7473b4880b957dcc92b14b with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e6949071-de22-48ca-880e-cfb67d9f9e70_isrForCrosstalkSources_1031717_7.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e6949071-de22-48ca-880e-cfb67d9f9e70_isrForCrosstalkSources_1031717_7.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 113f03e82e8fa709f2b63834e2fd7d48 with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/66883233-1f3b-4386-8a57-36265c0fa89b_isrForCrosstalkSources_1031717_11.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/66883233-1f3b-4386-8a57-36265c0fa89b_isrForCrosstalkSources_1031717_11.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b9641092cac1a5811b67b574793ac29 with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/27d4887c-ced7-4a44-8cc0-39377afdf279_isrForCrosstalkSources_1031708_53.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/27d4887c-ced7-4a44-8cc0-39377afdf279_isrForCrosstalkSources_1031708_53.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef346b30c782b104ab96a6068448ce38 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/553e1b15-2d0e-4048-9bac-ff20365e8250_isrForCrosstalkSources_1031704_54.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/553e1b15-2d0e-4048-9bac-ff20365e8250_isrForCrosstalkSources_1031704_54.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58b7e26934dc17cb58dedee8606c280b with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/ff9c20dd-582a-442e-b9e0-27a9358de369_isrForCrosstalkSources_1031706_8.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/ff9c20dd-582a-442e-b9e0-27a9358de369_isrForCrosstalkSources_1031706_8.stderr parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad328c84e889270d48a6d09475e1a5bf with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/b2bf4835-6b81-416c-908e-035772cbd953_isrForCrosstalkSources_1031735_9.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/b2bf4835-6b81-416c-908e-035772cbd953_isrForCrosstalkSources_1031735_9.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e8a597f333f25a8835af6069a49e7ba with result from task 411 2024-03-17 04:04:45 proc_lsst.multi:170 [INFO] [multi] job_id 17148020 proc_lsst.multi INFO: [multi] job_id 17148020 2024-03-17 04:04:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148020 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148020 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17148020 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-17 04:04:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-17 04:04:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1c1820ac-8197-40d0-a6b1-8e07d8ac3e92_isrForCrosstalkSources_1031732_45.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1c1820ac-8197-40d0-a6b1-8e07d8ac3e92_isrForCrosstalkSources_1031732_45.stderr proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50525d813ad4cf680a0600d67c0afbab with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/1dece97a-f403-4479-9698-8c8bf1316125_isrForCrosstalkSources_1031737_16.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/1dece97a-f403-4479-9698-8c8bf1316125_isrForCrosstalkSources_1031737_16.stderr parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdd38719e804b2c1c056dc70a05aad40 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/46f94bc4-360e-471c-8478-a28a30218c27_isrForCrosstalkSources_1031705_1.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/46f94bc4-360e-471c-8478-a28a30218c27_isrForCrosstalkSources_1031705_1.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50fc7cadb74518deb97f8c1629246f6f with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3128c055-1904-4984-b122-1ca6eacdd8cc_isrForCrosstalkSources_1031729_5.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3128c055-1904-4984-b122-1ca6eacdd8cc_isrForCrosstalkSources_1031729_5.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1eec5bff88feb17c2bbbdeaceb79a5c with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/042558c1-6b67-47d8-9187-1d400741c634_isrForCrosstalkSources_1031737_17.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/042558c1-6b67-47d8-9187-1d400741c634_isrForCrosstalkSources_1031737_17.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a29799f4e67e64be73abdef9aa9ded72 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/12391a4e-d888-45bf-ad3e-bad7a7fd96ef_isrForCrosstalkSources_1031726_28.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/12391a4e-d888-45bf-ad3e-bad7a7fd96ef_isrForCrosstalkSources_1031726_28.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 488de7f1fcf9296124bcd89421d48713 with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8e5a172d-2e77-4922-94a1-715480689bea_isrForCrosstalkSources_1031704_1.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8e5a172d-2e77-4922-94a1-715480689bea_isrForCrosstalkSources_1031704_1.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e952a39f7c9cf244c1cb0c55e9a7bdbc with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7eb9181a-2aca-4735-8242-4e10d4976b34_isrForCrosstalkSources_1031714_23.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7eb9181a-2aca-4735-8242-4e10d4976b34_isrForCrosstalkSources_1031714_23.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9587c6b9e7abf8f9074a65a6631c8fe0 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8f7267c9-970d-49ab-9850-8059da3fb7e4_isrForCrosstalkSources_1031704_56.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8f7267c9-970d-49ab-9850-8059da3fb7e4_isrForCrosstalkSources_1031704_56.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bff0eb002317bbe8475917db767a6a5 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/95ef499f-9dcc-4516-93ea-e82539f55cab_isrForCrosstalkSources_1031708_42.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/95ef499f-9dcc-4516-93ea-e82539f55cab_isrForCrosstalkSources_1031708_42.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1ae87b3224a3660e575e2756b92defa with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/397b9f52-f7ed-43fd-b71c-9d808b2f64f2_isrForCrosstalkSources_1031732_2.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/397b9f52-f7ed-43fd-b71c-9d808b2f64f2_isrForCrosstalkSources_1031732_2.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38e33c58f2d9b5c29ef6241885dc5577 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/974d8a0c-6291-40c1-a7c6-8585a16c5eed_isrForCrosstalkSources_1031712_6.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/974d8a0c-6291-40c1-a7c6-8585a16c5eed_isrForCrosstalkSources_1031712_6.stderr parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b15fa446f7ba0527a54466ac9e85cbe with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1d91ec54-4c6d-475c-875f-69f976334a03_isrForCrosstalkSources_1031733_2.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1d91ec54-4c6d-475c-875f-69f976334a03_isrForCrosstalkSources_1031733_2.stderr 2024-03-17 04:04:47 proc_lsst.multi:170 [INFO] [multi] job_id 17148021 proc_lsst.multi INFO: [multi] job_id 17148021 2024-03-17 04:04:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-17 04:04:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148021 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148021 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17148021 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-17 04:04:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-17 04:04:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-17 04:04:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c95d3fc5739af7affb3cc4289b33365 with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/29810860-129b-4371-a17b-4124b5182956_isrForCrosstalkSources_1031703_42.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/29810860-129b-4371-a17b-4124b5182956_isrForCrosstalkSources_1031703_42.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba8fe74bf81d20300fceec55d4bb93a7 with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f5304999-c6f1-4b4a-87f5-d3ba74a60ad4_isrForCrosstalkSources_1031705_7.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f5304999-c6f1-4b4a-87f5-d3ba74a60ad4_isrForCrosstalkSources_1031705_7.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd88d388137ccbf619ee599087cf17fe with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/837dbff7-a150-423b-a8a4-2315ac37f30b_isrForCrosstalkSources_1031731_31.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/837dbff7-a150-423b-a8a4-2315ac37f30b_isrForCrosstalkSources_1031731_31.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a17f022ca6134cc70cfd0e7d8a45706c with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/f994ced1-20f3-40d5-8140-76dc0dc29ea8_isrForCrosstalkSources_1031738_4.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/f994ced1-20f3-40d5-8140-76dc0dc29ea8_isrForCrosstalkSources_1031738_4.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e945f05d4a9f8c01763e32ebe64c8c13 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d6e34c3d-4a89-4e22-a905-ea6bc9a63364_isrForCrosstalkSources_1031730_60.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d6e34c3d-4a89-4e22-a905-ea6bc9a63364_isrForCrosstalkSources_1031730_60.stderr 2024-03-17 04:04:49 proc_lsst.multi:170 [INFO] [multi] job_id 17148022 proc_lsst.multi INFO: [multi] job_id 17148022 2024-03-17 04:04:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-17 04:04:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148022 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148022 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17148022 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-17 04:04:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-17 04:04:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-17 04:04:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f6a830ed31bb75441f41d5fa40f8506 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/70207f8c-1d52-48b0-a646-d58d0f695fd6_isrForCrosstalkSources_1031734_13.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/70207f8c-1d52-48b0-a646-d58d0f695fd6_isrForCrosstalkSources_1031734_13.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cc8ababfb47a10fdb25ead137d0614e with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3d16014f-8805-418a-81b5-61f6d9c6b92d_isrForCrosstalkSources_1031732_16.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3d16014f-8805-418a-81b5-61f6d9c6b92d_isrForCrosstalkSources_1031732_16.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5686fd07a1ecea0eb5b0aea4c62feea0 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/1e94e6d4-dfaa-498a-9d7d-4b0dd027593b_isrForCrosstalkSources_1031731_22.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/1e94e6d4-dfaa-498a-9d7d-4b0dd027593b_isrForCrosstalkSources_1031731_22.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfb1a8df04fa9784bfb5706a3249d6e5 with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/905a0aba-4066-4147-8027-260d73565b30_isrForCrosstalkSources_1031730_51.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/905a0aba-4066-4147-8027-260d73565b30_isrForCrosstalkSources_1031730_51.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50662fb283fad1425b8b5809468427e7 with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/94e9261b-abf2-413f-b6c9-eb473f619b98_isrForCrosstalkSources_1031717_23.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/94e9261b-abf2-413f-b6c9-eb473f619b98_isrForCrosstalkSources_1031717_23.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cad25c634e052543f98443958fbdcf70 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/de471be6-f7ea-423b-ae8e-d61e4fd67370_isrForCrosstalkSources_1031711_15.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/de471be6-f7ea-423b-ae8e-d61e4fd67370_isrForCrosstalkSources_1031711_15.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9e089b97419f5f007ff9d7a827bc065 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/9c854c7e-ee4f-4778-9fb7-c5524dc22930_isrForCrosstalkSources_1031719_30.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/9c854c7e-ee4f-4778-9fb7-c5524dc22930_isrForCrosstalkSources_1031719_30.stderr 2024-03-17 04:04:50 proc_lsst.multi:170 [INFO] [multi] job_id 17148023 proc_lsst.multi INFO: [multi] job_id 17148023 2024-03-17 04:04:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-17 04:04:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148023 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148023 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17148023 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-17 04:04:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-17 04:04:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-17 04:04:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d329e29ed92b77882839a44bae15640b with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e0a1e611-3397-46f2-ac25-0b11fffe6dbb_isrForCrosstalkSources_1031728_2.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e0a1e611-3397-46f2-ac25-0b11fffe6dbb_isrForCrosstalkSources_1031728_2.stderr parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aabbe740ab73f641825c4f8ff5afeb3f with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/daeb1dc8-f377-4b3d-8b59-591aaa012b67_isrForCrosstalkSources_1031731_1.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/daeb1dc8-f377-4b3d-8b59-591aaa012b67_isrForCrosstalkSources_1031731_1.stderr parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a58d531f6267c52355ceda9489a8275 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/f386a45a-d188-4f0a-b3bb-5753c3228c71_isrForCrosstalkSources_1031717_62.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/f386a45a-d188-4f0a-b3bb-5753c3228c71_isrForCrosstalkSources_1031717_62.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a2f98f92acb0bbb5641c8fc38c81275 with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/503faeed-9f51-4764-b1b8-a90d561fa9b6_isrForCrosstalkSources_1031707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/503faeed-9f51-4764-b1b8-a90d561fa9b6_isrForCrosstalkSources_1031707_61.stderr 2024-03-17 04:04:51 proc_lsst.multi:170 [INFO] [multi] job_id 17148024 proc_lsst.multi INFO: [multi] job_id 17148024 2024-03-17 04:04:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-17 04:04:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148024 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148024 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17148024 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-17 04:04:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-17 04:04:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-17 04:04:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa191b3b8b944f9638c87186848d43a9 with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/d8e99695-3bc8-4ce2-9c2c-c9486dc25e8d_isrForCrosstalkSources_1031705_35.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/d8e99695-3bc8-4ce2-9c2c-c9486dc25e8d_isrForCrosstalkSources_1031705_35.stderr parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fd258548b597fd90bca2d380f241704 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/15cb866a-b6cd-45a6-8213-d6009c15d862_isrForCrosstalkSources_1031727_62.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/15cb866a-b6cd-45a6-8213-d6009c15d862_isrForCrosstalkSources_1031727_62.stderr 2024-03-17 04:04:51 proc_lsst.multi:170 [INFO] [multi] job_id 17148025 proc_lsst.multi INFO: [multi] job_id 17148025 2024-03-17 04:04:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-17 04:04:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148025 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148025 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17148025 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-17 04:04:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-17 04:04:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-17 04:04:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ff12bb7c5b6544bee652028ca788ebf with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/b6510bb2-f96d-4b68-b136-626c7abe8a00_isrForCrosstalkSources_1031737_62.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/b6510bb2-f96d-4b68-b136-626c7abe8a00_isrForCrosstalkSources_1031737_62.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 187917ac28bf810c4f94cbd2bda2a291 with result from task 446 2024-03-17 04:04:52 proc_lsst.multi:170 [INFO] [multi] job_id 17148026 proc_lsst.multi INFO: [multi] job_id 17148026 2024-03-17 04:04:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-17 04:04:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148026 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148026 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17148026 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-17 04:04:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ea37b81f-d10a-43b9-ab4d-5c71eee7479d_isrForCrosstalkSources_1031737_23.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ea37b81f-d10a-43b9-ab4d-5c71eee7479d_isrForCrosstalkSources_1031737_23.stderr proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-17 04:04:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-17 04:04:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:04:53 proc_lsst.multi:170 [INFO] [multi] job_id 17148027 proc_lsst.multi INFO: [multi] job_id 17148027 2024-03-17 04:04:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-17 04:04:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148027 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148027 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17148027 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-17 04:04:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-17 04:04:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-17 04:04:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc21afeaff4dc19a60af74c0b22737b1 with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1765bdc2-1794-4cc7-9694-796eeaa7ff74_isrForCrosstalkSources_1031715_46.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1765bdc2-1794-4cc7-9694-796eeaa7ff74_isrForCrosstalkSources_1031715_46.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bef29c27803863dabb707373c9d06c4d with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/4d3caa72-92b8-4590-bbf1-76dae8401922_isrForCrosstalkSources_1031705_21.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/4d3caa72-92b8-4590-bbf1-76dae8401922_isrForCrosstalkSources_1031705_21.stderr parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941d60ce9a1698c404db9be1f9d8a45c with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/8762c009-2bee-4aaa-81cb-4ee17575302e_isrForCrosstalkSources_1031738_3.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/8762c009-2bee-4aaa-81cb-4ee17575302e_isrForCrosstalkSources_1031738_3.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffba587f6bcc1525a909938f8a8beb4a with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4111cc59-bd18-4588-8728-d43e4b1b794d_isrForCrosstalkSources_1031699_17.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4111cc59-bd18-4588-8728-d43e4b1b794d_isrForCrosstalkSources_1031699_17.stderr parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 596dbde70c87a474f12f6278d470c978 with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/9bb5ca01-d13e-4bce-b856-5e04b8a6668b_isrForCrosstalkSources_1031710_10.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/9bb5ca01-d13e-4bce-b856-5e04b8a6668b_isrForCrosstalkSources_1031710_10.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 026713aef59909fa38348ac94913099f with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a328aeb7-4554-4c70-bb11-4960509a0189_isrForCrosstalkSources_1031726_33.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a328aeb7-4554-4c70-bb11-4960509a0189_isrForCrosstalkSources_1031726_33.stderr parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7026d02efd217ba240a1ca47d3e052c0 with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/912c2a58-f1f7-45e4-9e98-ebc6968c35f0_isrForCrosstalkSources_1031711_33.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/912c2a58-f1f7-45e4-9e98-ebc6968c35f0_isrForCrosstalkSources_1031711_33.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31bd4111c0f8b0004b01b2b52bb502c9 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/86b33649-ad5d-4c9e-a78d-139c0517c014_isrForCrosstalkSources_1031721_24.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/86b33649-ad5d-4c9e-a78d-139c0517c014_isrForCrosstalkSources_1031721_24.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7665966965d6ed189c8c1c912c951122 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bb4656ed-de8f-4b48-8a7e-7143c244cab0_isrForCrosstalkSources_1031729_15.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bb4656ed-de8f-4b48-8a7e-7143c244cab0_isrForCrosstalkSources_1031729_15.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eefadfbc783642f4b58f321170a48001 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/cc1f9671-19ec-45fc-9e36-aca93bae5529_isrForCrosstalkSources_1031737_5.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/cc1f9671-19ec-45fc-9e36-aca93bae5529_isrForCrosstalkSources_1031737_5.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bfd5ec19941ae9ef5867df0dfce0c44 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cb45e38d-a9cc-40b6-8848-5a6ca8bd7f5c_isrForCrosstalkSources_1031707_9.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cb45e38d-a9cc-40b6-8848-5a6ca8bd7f5c_isrForCrosstalkSources_1031707_9.stderr parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 213a3e8ea1f47a4a4565da240b9ea08f with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ad2d41c9-ff7c-4a01-b76a-b1f9e7715582_isrForCrosstalkSources_1031725_37.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ad2d41c9-ff7c-4a01-b76a-b1f9e7715582_isrForCrosstalkSources_1031725_37.stderr parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 537c7ca48d9f461fbb9b6d1491aab071 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/97666d4e-d27c-44a5-87be-47475d887fd7_isrForCrosstalkSources_1031730_59.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/97666d4e-d27c-44a5-87be-47475d887fd7_isrForCrosstalkSources_1031730_59.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5abc84218bfe2e162ea8f0aaa5c7ae95 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/112cf319-cfc5-4d7f-8fcf-1c96864999ba_isrForCrosstalkSources_1031733_38.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/112cf319-cfc5-4d7f-8fcf-1c96864999ba_isrForCrosstalkSources_1031733_38.stderr 2024-03-17 04:04:54 proc_lsst.multi:170 [INFO] [multi] job_id 17148028 proc_lsst.multi INFO: [multi] job_id 17148028 2024-03-17 04:04:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-17 04:04:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148028 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148028 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17148028 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-17 04:04:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-17 04:04:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-17 04:04:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1e0d5419793af7b3ee3286a7c5a1879 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/a6258bc9-cbdf-40f4-916b-d397111c3e41_isrForCrosstalkSources_1031733_44.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/a6258bc9-cbdf-40f4-916b-d397111c3e41_isrForCrosstalkSources_1031733_44.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac2f91196a11e858b197a4e812012367 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/9c03017d-cc9e-4c14-b456-8b9a4394c594_isrForCrosstalkSources_1031738_15.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/9c03017d-cc9e-4c14-b456-8b9a4394c594_isrForCrosstalkSources_1031738_15.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ea3fd46edf7ef3b2753759f48bdd729 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/f8903d6f-2d29-4037-86fa-98fec552c472_isrForCrosstalkSources_1031704_8.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/f8903d6f-2d29-4037-86fa-98fec552c472_isrForCrosstalkSources_1031704_8.stderr parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fd626fc50ac8cd09d62b44b93d2c279 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3218ab54-7ef5-478e-ace6-a4c67284e9e3_isrForCrosstalkSources_1031710_32.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3218ab54-7ef5-478e-ace6-a4c67284e9e3_isrForCrosstalkSources_1031710_32.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a50bf8b43f148eac8dea16713c8c6654 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2ddd0b38-1851-4980-ac9c-48bca0f37a79_isrForCrosstalkSources_1031726_14.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2ddd0b38-1851-4980-ac9c-48bca0f37a79_isrForCrosstalkSources_1031726_14.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bed328bf106b4ad50803ae0f18d8def8 with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a2abeaba-8387-403d-88b9-b557983543fa_isrForCrosstalkSources_1031730_49.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a2abeaba-8387-403d-88b9-b557983543fa_isrForCrosstalkSources_1031730_49.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4267f3b71b946651f4c1452e7edea181 with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7bcbfe50-e411-459e-bb9f-0fd52acf78c4_isrForCrosstalkSources_1031735_10.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7bcbfe50-e411-459e-bb9f-0fd52acf78c4_isrForCrosstalkSources_1031735_10.stderr parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e044abd2955a6f728b88f40030f5972 with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/83704301-57b8-482d-9fe6-7e93804104bf_isrForCrosstalkSources_1031718_16.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/83704301-57b8-482d-9fe6-7e93804104bf_isrForCrosstalkSources_1031718_16.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0d109a66502f5737686060bfc730267 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e793c485-16b1-4afb-8385-297a67bfae4a_isrForCrosstalkSources_1031731_39.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e793c485-16b1-4afb-8385-297a67bfae4a_isrForCrosstalkSources_1031731_39.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fa8dc780370f822fa0b8d39e407ad0e with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/85a9bdf3-2db2-4e16-9b0a-63b82470d58f_isrForCrosstalkSources_1031735_15.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/85a9bdf3-2db2-4e16-9b0a-63b82470d58f_isrForCrosstalkSources_1031735_15.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4790e34fed198a0ca0ef24de0831624 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/db71bdd1-05f3-40fa-9bab-3767b2015e63_isrForCrosstalkSources_1031709_8.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/db71bdd1-05f3-40fa-9bab-3767b2015e63_isrForCrosstalkSources_1031709_8.stderr parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d109fb8d8136b45a6dbf909b492cd90 with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/a6947acd-3746-4a62-bdff-01a3bf4d1319_isrForCrosstalkSources_1031727_10.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/a6947acd-3746-4a62-bdff-01a3bf4d1319_isrForCrosstalkSources_1031727_10.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0487d818b7ee6d923a67a9d2ba97b0b with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/48f03e26-bcb6-44db-b701-378aa93aff66_isrForCrosstalkSources_1031722_40.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/48f03e26-bcb6-44db-b701-378aa93aff66_isrForCrosstalkSources_1031722_40.stderr 2024-03-17 04:04:56 proc_lsst.multi:170 [INFO] [multi] job_id 17148029 proc_lsst.multi INFO: [multi] job_id 17148029 2024-03-17 04:04:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-17 04:04:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148029 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148029 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17148029 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-17 04:04:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-17 04:04:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-17 04:04:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ca3dcce5183640f7dc5e7a3bd7f34e5 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3aa42ae6-5be0-44f7-bb3a-1748785b1e5f_isrForCrosstalkSources_1031735_3.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3aa42ae6-5be0-44f7-bb3a-1748785b1e5f_isrForCrosstalkSources_1031735_3.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a1113c0422c7fab50fde3a9ea797264 with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d499498c-d0e6-4ccb-9464-9399bc011fff_isrForCrosstalkSources_1031722_17.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d499498c-d0e6-4ccb-9464-9399bc011fff_isrForCrosstalkSources_1031722_17.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbc11ff3ef9e29ba46025e232fbc3fed with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/9d23937a-c75e-4a98-b72a-5f0abd2530c4_isrForCrosstalkSources_1031714_52.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/9d23937a-c75e-4a98-b72a-5f0abd2530c4_isrForCrosstalkSources_1031714_52.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97569b1ab6628ca971b3f1e40420455a with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/2c1ab183-ab9c-459f-b3b9-f6d5f441f97a_isrForCrosstalkSources_1031730_25.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/2c1ab183-ab9c-459f-b3b9-f6d5f441f97a_isrForCrosstalkSources_1031730_25.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91f6f4f5c7b172cde9917d0bf2f307a9 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7706f961-e9a5-4779-8764-b4a8eaf76a99_isrForCrosstalkSources_1031721_25.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7706f961-e9a5-4779-8764-b4a8eaf76a99_isrForCrosstalkSources_1031721_25.stderr parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 010884bb5471835704f94c3e74ea4311 with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2fa0b349-9ef8-4a68-921b-0cd87c3db342_isrForCrosstalkSources_1031727_32.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2fa0b349-9ef8-4a68-921b-0cd87c3db342_isrForCrosstalkSources_1031727_32.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 226beb9181280e83f68ae4cfaff1c7d7 with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cb45c076-8aad-4a6e-b6c4-794733e2d2c0_isrForCrosstalkSources_1031707_42.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/cb45c076-8aad-4a6e-b6c4-794733e2d2c0_isrForCrosstalkSources_1031707_42.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ce612c1172c617ba9660195abc4c6c7 with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c23e5d1b-af57-4a38-94b1-cb655c1a5432_isrForCrosstalkSources_1031726_55.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c23e5d1b-af57-4a38-94b1-cb655c1a5432_isrForCrosstalkSources_1031726_55.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aab7db67e82658e6c33472913729f616 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/1227ccb6-1fb7-43a8-9cc2-5c3fe779c957_isrForCrosstalkSources_1031714_48.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/1227ccb6-1fb7-43a8-9cc2-5c3fe779c957_isrForCrosstalkSources_1031714_48.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f21d993b5b9f27bd541fc3601babb74b with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/252ee987-19bd-4cd4-b1d0-3dccb238df32_isrForCrosstalkSources_1031733_55.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/252ee987-19bd-4cd4-b1d0-3dccb238df32_isrForCrosstalkSources_1031733_55.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adbcfd57b4645aaeb256155e89e21bb6 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/eb9c002b-cb02-4919-864a-c118dc889c5a_isrForCrosstalkSources_1031718_33.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/eb9c002b-cb02-4919-864a-c118dc889c5a_isrForCrosstalkSources_1031718_33.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7500828a9c50b4e86ee64d5db0c70bc8 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/40f6f468-39bc-4984-9fc8-770089a0f929_isrForCrosstalkSources_1031713_7.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/40f6f468-39bc-4984-9fc8-770089a0f929_isrForCrosstalkSources_1031713_7.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 048175279618206f8c1c319a4375db4f with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/84750fd8-82c4-4a76-adef-9dc9a64ef17c_isrForCrosstalkSources_1031736_40.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/84750fd8-82c4-4a76-adef-9dc9a64ef17c_isrForCrosstalkSources_1031736_40.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36444d9bc4a0c307c2f81cab89a608ea with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/5ef3d934-4ae5-4199-a4da-c8850da5f87b_isrForCrosstalkSources_1031699_53.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/5ef3d934-4ae5-4199-a4da-c8850da5f87b_isrForCrosstalkSources_1031699_53.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f29705e003aff35d3f0fa29cd33b6d36 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/a9cc6abb-01e8-464d-81cd-e6a2a532d2bc_isrForCrosstalkSources_1031732_27.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/a9cc6abb-01e8-464d-81cd-e6a2a532d2bc_isrForCrosstalkSources_1031732_27.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8e35855e50fd00e62ef4cd78c88fb03 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d7aa2d4b-9238-44e5-99f5-019764231df5_isrForCrosstalkSources_1031703_46.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d7aa2d4b-9238-44e5-99f5-019764231df5_isrForCrosstalkSources_1031703_46.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ed0bf8bce21621538466e68fa8d2b5e with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7728ed00-1ec8-45f6-8430-b43ae6165584_isrForCrosstalkSources_1031721_60.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7728ed00-1ec8-45f6-8430-b43ae6165584_isrForCrosstalkSources_1031721_60.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64ec2059c3719a91b6b5df77cd8f152e with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/59261fd3-1b45-48d3-af3e-dda86af19ea9_isrForCrosstalkSources_1031714_38.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/59261fd3-1b45-48d3-af3e-dda86af19ea9_isrForCrosstalkSources_1031714_38.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 459b4ee6d896e97e297314ef487ec6a5 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8b2d1ecb-4b61-46d4-b296-57bd12186f16_isrForCrosstalkSources_1031733_59.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8b2d1ecb-4b61-46d4-b296-57bd12186f16_isrForCrosstalkSources_1031733_59.stderr 2024-03-17 04:04:59 proc_lsst.multi:170 [INFO] [multi] job_id 17148030 proc_lsst.multi INFO: [multi] job_id 17148030 2024-03-17 04:04:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148030 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148030 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17148030 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-17 04:04:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-17 04:04:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:04:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:04:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:04:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:04:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:04:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:04:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:04:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:04:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:04:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:04:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:04:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baa2cf532f4eee6d99cfa8e7c4014a8e with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/ba621ef7-791c-42d0-b3e5-2d3790036eeb_isrForCrosstalkSources_1031733_48.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/ba621ef7-791c-42d0-b3e5-2d3790036eeb_isrForCrosstalkSources_1031733_48.stderr parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96ad0b2a4b0bd249071f544c7e57d8fb with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5e4b08ad-9df0-43ce-98a2-70d4c938a7cb_isrForCrosstalkSources_1031736_1.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5e4b08ad-9df0-43ce-98a2-70d4c938a7cb_isrForCrosstalkSources_1031736_1.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 572f1e9ac81fac6b47d0ad1999f540a1 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1cee8586-8e6a-4661-bdff-33a3193f136c_isrForCrosstalkSources_1031715_36.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1cee8586-8e6a-4661-bdff-33a3193f136c_isrForCrosstalkSources_1031715_36.stderr parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72990ec6d1fbbc9c9b7620078386c867 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/aac93c07-3601-44f8-93e3-97abd5511e3d_isrForCrosstalkSources_1031702_6.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/aac93c07-3601-44f8-93e3-97abd5511e3d_isrForCrosstalkSources_1031702_6.stderr parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f48a2e12599d5a9359d178585503865 with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b3a55dd0-7dbf-401c-9a28-a92577fda5a6_isrForCrosstalkSources_1031708_37.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b3a55dd0-7dbf-401c-9a28-a92577fda5a6_isrForCrosstalkSources_1031708_37.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94817f11475b5b9be8176b358410d252 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/9d1ba715-34c9-477f-9b14-7ee8e05a0986_isrForCrosstalkSources_1031701_58.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/9d1ba715-34c9-477f-9b14-7ee8e05a0986_isrForCrosstalkSources_1031701_58.stderr 2024-03-17 04:05:00 proc_lsst.multi:170 [INFO] [multi] job_id 17148031 proc_lsst.multi INFO: [multi] job_id 17148031 2024-03-17 04:05:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-17 04:05:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148031 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148031 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17148031 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-17 04:05:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-17 04:05:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-17 04:05:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b6893e533af5f01eea31fd1d07c3721 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a1ccae7d-e93f-4d0c-b205-866d9e62be37_isrForCrosstalkSources_1031718_45.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a1ccae7d-e93f-4d0c-b205-866d9e62be37_isrForCrosstalkSources_1031718_45.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c00baefb3dfa000137951fdca088065 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/db7f0be4-57da-4612-89ab-3c1bff6e95c6_isrForCrosstalkSources_1031725_56.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/db7f0be4-57da-4612-89ab-3c1bff6e95c6_isrForCrosstalkSources_1031725_56.stderr parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9d7933d4b0fd76489c28d004ad0b6dd with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f23c306c-5ecc-4a66-8027-782a16d9931b_isrForCrosstalkSources_1031737_39.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f23c306c-5ecc-4a66-8027-782a16d9931b_isrForCrosstalkSources_1031737_39.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac81b9113a9388442cb9a52b8aaefc2f with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5793b370-46bf-423d-9b84-958fa5db5497_isrForCrosstalkSources_1031705_14.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5793b370-46bf-423d-9b84-958fa5db5497_isrForCrosstalkSources_1031705_14.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6e25b9fe8a676369521bd3176f7c1d1 with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/53e29225-e91b-4969-85f3-cb107f175b31_isrForCrosstalkSources_1031734_41.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/53e29225-e91b-4969-85f3-cb107f175b31_isrForCrosstalkSources_1031734_41.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 599becb359ff15356a925985d03612b7 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4699e5ce-70e7-4911-84c2-08ba672c9e11_isrForCrosstalkSources_1031706_19.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4699e5ce-70e7-4911-84c2-08ba672c9e11_isrForCrosstalkSources_1031706_19.stderr parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db22765507179c2dafdd6ff93ef814bb with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/7871b96c-e5d8-4111-b88a-af53b1e3fdf0_isrForCrosstalkSources_1031711_39.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/7871b96c-e5d8-4111-b88a-af53b1e3fdf0_isrForCrosstalkSources_1031711_39.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 030019fd361b89a211813ba2f4dd475c with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/97f68eee-37a1-4e7b-b376-2287a714d29a_isrForCrosstalkSources_1031711_4.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/97f68eee-37a1-4e7b-b376-2287a714d29a_isrForCrosstalkSources_1031711_4.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b46c0736b2bb685a4296c7da40537b72 with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a5ea8139-636c-41f4-833d-5537f13c16d1_isrForCrosstalkSources_1031738_14.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a5ea8139-636c-41f4-833d-5537f13c16d1_isrForCrosstalkSources_1031738_14.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47317961632a3fc01d4630b2103edfaa with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0760b9cd-0ebc-4823-9930-b0febb6e658d_isrForCrosstalkSources_1031720_62.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0760b9cd-0ebc-4823-9930-b0febb6e658d_isrForCrosstalkSources_1031720_62.stderr parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19a1f9875c339f2405342186a66871c4 with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/c177d6f0-2640-43cd-af41-4ac4ad5767f8_isrForCrosstalkSources_1031735_42.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/c177d6f0-2640-43cd-af41-4ac4ad5767f8_isrForCrosstalkSources_1031735_42.stderr 2024-03-17 04:05:03 proc_lsst.multi:170 [INFO] [multi] job_id 17148032 proc_lsst.multi INFO: [multi] job_id 17148032 2024-03-17 04:05:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-17 04:05:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148032 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148032 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17148032 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-17 04:05:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-17 04:05:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-17 04:05:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 088be356b187bec0314d84e66c00d641 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5fe09a30-8604-488a-b3e6-2ae88ea2e213_isrForCrosstalkSources_1031720_61.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5fe09a30-8604-488a-b3e6-2ae88ea2e213_isrForCrosstalkSources_1031720_61.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df1741fad637363eff777595520cd33f with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/d518331a-5af2-4267-b959-931f80b4d335_isrForCrosstalkSources_1031700_15.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/d518331a-5af2-4267-b959-931f80b4d335_isrForCrosstalkSources_1031700_15.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccb6bb106966382da7675d4dde7d5533 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/1085ba3b-f4f3-4470-99d4-df84f24b435e_isrForCrosstalkSources_1031707_28.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/1085ba3b-f4f3-4470-99d4-df84f24b435e_isrForCrosstalkSources_1031707_28.stderr parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7b072377ab60df156c43522d62cf20b with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e3e611a5-c23c-4d7c-a898-16d6a16dba8c_isrForCrosstalkSources_1031717_1.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e3e611a5-c23c-4d7c-a898-16d6a16dba8c_isrForCrosstalkSources_1031717_1.stderr parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7712dfc78e3d84b415112a5bcc7759b with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/24c8cd32-85f1-4f4c-989e-5eff645aec73_isrForCrosstalkSources_1031719_3.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/24c8cd32-85f1-4f4c-989e-5eff645aec73_isrForCrosstalkSources_1031719_3.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5074a8763f173251b42a46797a7666e4 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/345b8153-8b1a-4f08-85bb-34b35a1c0e3d_isrForCrosstalkSources_1031704_38.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/345b8153-8b1a-4f08-85bb-34b35a1c0e3d_isrForCrosstalkSources_1031704_38.stderr parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1652286a86a28f31c7ccf31298e4e238 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a030ee6d-c073-423e-9b42-69c1ec14b48c_isrForCrosstalkSources_1031723_38.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a030ee6d-c073-423e-9b42-69c1ec14b48c_isrForCrosstalkSources_1031723_38.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 692d18f0761a6bc718801b80a1d93469 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/562a4569-b67e-4428-910d-ba819ac56593_isrForCrosstalkSources_1031699_4.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/562a4569-b67e-4428-910d-ba819ac56593_isrForCrosstalkSources_1031699_4.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71c6a1054e7429804d2bd006fabef7bb with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ef69c233-6ed4-455e-ba8e-a61705e75bb2_isrForCrosstalkSources_1031717_56.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ef69c233-6ed4-455e-ba8e-a61705e75bb2_isrForCrosstalkSources_1031717_56.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcef725091cb7ebdaada7fe8184c47d6 with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0521acf4-0607-4058-bcfb-9b172579c44a_isrForCrosstalkSources_1031705_18.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0521acf4-0607-4058-bcfb-9b172579c44a_isrForCrosstalkSources_1031705_18.stderr parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef06322f36448852fbc40f7af86fbd9a with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e9b9bc7f-45d9-47b4-9e76-ec28f8c00908_isrForCrosstalkSources_1031716_31.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e9b9bc7f-45d9-47b4-9e76-ec28f8c00908_isrForCrosstalkSources_1031716_31.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22b3778195d4360f027f464f30655480 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7f02ce67-0da2-43b1-ab1c-28b26123a90c_isrForCrosstalkSources_1031721_49.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7f02ce67-0da2-43b1-ab1c-28b26123a90c_isrForCrosstalkSources_1031721_49.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e72e66473e823a52b16f43eec3ee139 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/5b19da89-caa0-4dbe-8593-e8bdf9cc98e3_isrForCrosstalkSources_1031713_35.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/5b19da89-caa0-4dbe-8593-e8bdf9cc98e3_isrForCrosstalkSources_1031713_35.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3d8aeb7778ffd2e99c11fc27c071835 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/75d8cbf0-6d39-4b41-978d-5f30c84c2cc6_isrForCrosstalkSources_1031702_9.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/75d8cbf0-6d39-4b41-978d-5f30c84c2cc6_isrForCrosstalkSources_1031702_9.stderr parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3424115bfc0154ab84bea18e8567f901 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/e9098657-29d2-419f-a1dc-4da55d489999_isrForCrosstalkSources_1031737_46.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/e9098657-29d2-419f-a1dc-4da55d489999_isrForCrosstalkSources_1031737_46.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8b7876b5706262975eaf6ed9bd41fe0 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a250396b-070c-4e46-a0d9-2f6e23c31492_isrForCrosstalkSources_1031736_22.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a250396b-070c-4e46-a0d9-2f6e23c31492_isrForCrosstalkSources_1031736_22.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd74dca1e51106bbfbcc358c9c5ab795 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/072719b1-30ba-4e6a-81f5-60d73952935e_isrForCrosstalkSources_1031719_35.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/072719b1-30ba-4e6a-81f5-60d73952935e_isrForCrosstalkSources_1031719_35.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 845318d88c1eb3522595df500626ee19 with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d53faf69-6c41-4a43-8d15-4314e8a75442_isrForCrosstalkSources_1031727_44.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d53faf69-6c41-4a43-8d15-4314e8a75442_isrForCrosstalkSources_1031727_44.stderr parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c82beb2b872e28dc41204046a8e8d00 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e5a3309b-7de1-4297-b95e-78d02e89d177_isrForCrosstalkSources_1031709_58.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e5a3309b-7de1-4297-b95e-78d02e89d177_isrForCrosstalkSources_1031709_58.stderr 2024-03-17 04:05:05 proc_lsst.multi:170 [INFO] [multi] job_id 17148033 proc_lsst.multi INFO: [multi] job_id 17148033 2024-03-17 04:05:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-17 04:05:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148033 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148033 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17148033 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-17 04:05:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-17 04:05:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-17 04:05:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ecda23bf51c77ebc447a9fd53f9d800 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/19a58c1c-6a36-49c6-9ecf-16812b0d5899_isrForCrosstalkSources_1031715_60.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/19a58c1c-6a36-49c6-9ecf-16812b0d5899_isrForCrosstalkSources_1031715_60.stderr 2024-03-17 04:05:06 proc_lsst.multi:170 [INFO] [multi] job_id 17148034 proc_lsst.multi INFO: [multi] job_id 17148034 2024-03-17 04:05:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-17 04:05:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148034 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148034 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17148034 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-17 04:05:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-17 04:05:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-17 04:05:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f1d022735e1a49f21cd229d6fdc2a4 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/88469836-5683-427e-80d6-cd05f50612c4_isrForCrosstalkSources_1031722_25.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/88469836-5683-427e-80d6-cd05f50612c4_isrForCrosstalkSources_1031722_25.stderr parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8310487d7fb09762236683033da4240 with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/0485c8f5-40ae-4f17-8f1f-d823fbd4ee79_isrForCrosstalkSources_1031702_32.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/0485c8f5-40ae-4f17-8f1f-d823fbd4ee79_isrForCrosstalkSources_1031702_32.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2db47fd84b39cd1b193a064b021a2d95 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c535190b-69c0-48ea-92eb-f5e7e0db1f8a_isrForCrosstalkSources_1031734_14.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c535190b-69c0-48ea-92eb-f5e7e0db1f8a_isrForCrosstalkSources_1031734_14.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22d55a6d2caff7e9aa9806cffa52865c with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/41b0f51c-4ed7-45b3-9053-8768712bf953_isrForCrosstalkSources_1031711_45.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/41b0f51c-4ed7-45b3-9053-8768712bf953_isrForCrosstalkSources_1031711_45.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6b6f9b680f8aaeff259306cfbadd384 with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/e2f2697f-7783-4705-9047-537713726ec0_isrForCrosstalkSources_1031715_25.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/e2f2697f-7783-4705-9047-537713726ec0_isrForCrosstalkSources_1031715_25.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99e3d4ab5a922d88726857af19f80dd9 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ec97510c-af9e-4266-8c3b-66a5d6349363_isrForCrosstalkSources_1031738_16.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ec97510c-af9e-4266-8c3b-66a5d6349363_isrForCrosstalkSources_1031738_16.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96a08f962bc86164e14ddf631cf0ea4b with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e866cba7-ba0b-4fee-9d5e-1c696b81c748_isrForCrosstalkSources_1031733_17.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e866cba7-ba0b-4fee-9d5e-1c696b81c748_isrForCrosstalkSources_1031733_17.stderr 2024-03-17 04:05:07 proc_lsst.multi:170 [INFO] [multi] job_id 17148035 proc_lsst.multi INFO: [multi] job_id 17148035 2024-03-17 04:05:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-17 04:05:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148035 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148035 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17148035 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-17 04:05:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-17 04:05:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-17 04:05:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0870d8ca6971d5240223dcc2755a561b with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/5053c429-107d-49d7-979a-949d1dbeb840_isrForCrosstalkSources_1031710_8.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/5053c429-107d-49d7-979a-949d1dbeb840_isrForCrosstalkSources_1031710_8.stderr 2024-03-17 04:05:08 proc_lsst.multi:170 [INFO] [multi] job_id 17148036 proc_lsst.multi INFO: [multi] job_id 17148036 2024-03-17 04:05:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-17 04:05:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148036 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148036 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17148036 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-17 04:05:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-17 04:05:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-17 04:05:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:05:08 proc_lsst.multi:170 [INFO] [multi] job_id 17148037 proc_lsst.multi INFO: [multi] job_id 17148037 2024-03-17 04:05:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-17 04:05:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148037 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148037 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17148037 parsl.executors.status_handling INFO: Allocated block ID 93 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-93 2024-03-17 04:05:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 2024-03-17 04:05:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-17 04:05:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 092414497ecefb1fcd570a75348ec66d with result from task 593 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/971a7ad1-44d4-4725-b516-4f993e7e8e01_isrForCrosstalkSources_1031722_15.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/971a7ad1-44d4-4725-b516-4f993e7e8e01_isrForCrosstalkSources_1031722_15.stderr 2024-03-17 04:05:09 proc_lsst.multi:170 [INFO] [multi] job_id 17148038 proc_lsst.multi INFO: [multi] job_id 17148038 2024-03-17 04:05:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-17 04:05:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148038 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148038 parsl.executors.status_handling DEBUG: Launched block 93 on executor multi with job ID 17148038 parsl.executors.status_handling INFO: Allocated block ID 94 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-94 2024-03-17 04:05:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 2024-03-17 04:05:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-17 04:05:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd819119c77030c7a643fa615cd09a8a with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/47df78b0-2e3b-4f88-b68c-4fd73a600ac0_isrForCrosstalkSources_1031699_5.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/47df78b0-2e3b-4f88-b68c-4fd73a600ac0_isrForCrosstalkSources_1031699_5.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03ad687d38d61a3b9bc1d8ed83ed040f with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/099dad65-b159-4f37-b3f8-d71a1bac4d84_isrForCrosstalkSources_1031725_25.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/099dad65-b159-4f37-b3f8-d71a1bac4d84_isrForCrosstalkSources_1031725_25.stderr 2024-03-17 04:05:10 proc_lsst.multi:170 [INFO] [multi] job_id 17148039 proc_lsst.multi INFO: [multi] job_id 17148039 2024-03-17 04:05:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-17 04:05:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148039 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148039 parsl.executors.status_handling DEBUG: Launched block 94 on executor multi with job ID 17148039 parsl.executors.status_handling INFO: Allocated block ID 95 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-95 2024-03-17 04:05:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 2024-03-17 04:05:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-17 04:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca22ea2d2e0ae931def6a5642b969794 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/15a231dc-e121-44e2-80d7-b5b551093ff9_isrForCrosstalkSources_1031711_32.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/15a231dc-e121-44e2-80d7-b5b551093ff9_isrForCrosstalkSources_1031711_32.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f24bac24ef0d1e14df1443db98966e1 with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/d7ead2b4-b700-43bb-9ee5-44a7a73b6909_isrForCrosstalkSources_1031700_29.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/d7ead2b4-b700-43bb-9ee5-44a7a73b6909_isrForCrosstalkSources_1031700_29.stderr parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cb490cba0c17443ff1b6edefad66b29 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a045939d-1940-4381-a3ba-0479cb34caf9_isrForCrosstalkSources_1031722_13.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a045939d-1940-4381-a3ba-0479cb34caf9_isrForCrosstalkSources_1031722_13.stderr 2024-03-17 04:05:10 proc_lsst.multi:170 [INFO] [multi] job_id 17148040 proc_lsst.multi INFO: [multi] job_id 17148040 2024-03-17 04:05:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-17 04:05:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148040 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148040 parsl.executors.status_handling DEBUG: Launched block 95 on executor multi with job ID 17148040 parsl.executors.status_handling INFO: Allocated block ID 96 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-96 2024-03-17 04:05:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 2024-03-17 04:05:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-17 04:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b50ac2a1c271e2e3f8bf877829588b44 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/837f3e37-e753-412f-ada8-8e89f2a59da2_isrForCrosstalkSources_1031704_46.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/837f3e37-e753-412f-ada8-8e89f2a59da2_isrForCrosstalkSources_1031704_46.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5eacdbf9a3d80f889774cc105ec12b9 with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/d58e91f0-05c3-416e-8e34-432564f45b92_isrForCrosstalkSources_1031721_48.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/d58e91f0-05c3-416e-8e34-432564f45b92_isrForCrosstalkSources_1031721_48.stderr 2024-03-17 04:05:11 proc_lsst.multi:170 [INFO] [multi] job_id 17148041 proc_lsst.multi INFO: [multi] job_id 17148041 2024-03-17 04:05:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-17 04:05:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148041 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148041 parsl.executors.status_handling DEBUG: Launched block 96 on executor multi with job ID 17148041 parsl.executors.status_handling INFO: Allocated block ID 97 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-97 2024-03-17 04:05:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 2024-03-17 04:05:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-17 04:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2a09b9ddfae36f1682f109fb78144ee with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8f79e1ff-44a4-475d-a35d-d3d0c466ec39_isrForCrosstalkSources_1031709_21.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8f79e1ff-44a4-475d-a35d-d3d0c466ec39_isrForCrosstalkSources_1031709_21.stderr parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b499802050128191eccfee2bc19f0a03 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/02fa0833-3fe8-46e3-8ebc-efaad8076fbc_isrForCrosstalkSources_1031726_7.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/02fa0833-3fe8-46e3-8ebc-efaad8076fbc_isrForCrosstalkSources_1031726_7.stderr 2024-03-17 04:05:12 proc_lsst.multi:170 [INFO] [multi] job_id 17148042 proc_lsst.multi INFO: [multi] job_id 17148042 2024-03-17 04:05:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-17 04:05:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148042 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148042 parsl.executors.status_handling DEBUG: Launched block 97 on executor multi with job ID 17148042 parsl.executors.status_handling INFO: Allocated block ID 98 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-98 2024-03-17 04:05:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 2024-03-17 04:05:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-17 04:05:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a0f23d157e56e077e99c53dfa4b5741 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b6a80008-2956-417c-8936-600ec0affb43_isrForCrosstalkSources_1031714_9.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b6a80008-2956-417c-8936-600ec0affb43_isrForCrosstalkSources_1031714_9.stderr 2024-03-17 04:05:13 proc_lsst.multi:170 [INFO] [multi] job_id 17148043 proc_lsst.multi INFO: [multi] job_id 17148043 2024-03-17 04:05:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-17 04:05:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148043 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148043 parsl.executors.status_handling DEBUG: Launched block 98 on executor multi with job ID 17148043 parsl.executors.status_handling INFO: Allocated block ID 99 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-99 2024-03-17 04:05:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 2024-03-17 04:05:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-17 04:05:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97164b37b2e92946d7576059fcd1450e with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ea3f8cc6-2c4f-4ef9-adf5-5bdc1ff06f39_isrForCrosstalkSources_1031713_47.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ea3f8cc6-2c4f-4ef9-adf5-5bdc1ff06f39_isrForCrosstalkSources_1031713_47.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4301aad71e4417ec4e30526653314649 with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a545e35e-958c-4fdb-a519-d2fc21eca840_isrForCrosstalkSources_1031706_60.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a545e35e-958c-4fdb-a519-d2fc21eca840_isrForCrosstalkSources_1031706_60.stderr parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd6240da7504ccfe3c0e5c349a3aa59f with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0041d2f2-8fcb-40d7-9146-2a39346a1f61_isrForCrosstalkSources_1031719_20.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0041d2f2-8fcb-40d7-9146-2a39346a1f61_isrForCrosstalkSources_1031719_20.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03fbb2b31171a8be06f97d6f76ecccdf with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/acbaf117-9f31-41b3-923d-4f438efed85b_isrForCrosstalkSources_1031731_33.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/acbaf117-9f31-41b3-923d-4f438efed85b_isrForCrosstalkSources_1031731_33.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39b444742c108b3b3f34d190409af5c3 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/58dd09dd-08eb-491f-ae36-98912e475d12_isrForCrosstalkSources_1031709_34.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/58dd09dd-08eb-491f-ae36-98912e475d12_isrForCrosstalkSources_1031709_34.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7e66a07dbbd07f5c4101b58a0d88dcf with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0e644181-e901-4baa-ae58-15d5e91da8c2_isrForCrosstalkSources_1031724_22.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0e644181-e901-4baa-ae58-15d5e91da8c2_isrForCrosstalkSources_1031724_22.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1bf0e3dce7a835b74f81e7c15a1ba7d with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9a570f92-c71e-4ad5-84ec-53287165f0e8_isrForCrosstalkSources_1031730_24.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9a570f92-c71e-4ad5-84ec-53287165f0e8_isrForCrosstalkSources_1031730_24.stderr parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50bc0661b61683426416d48f27b03075 with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3c04595b-bffe-4c03-9a37-222a7f2aa7bd_isrForCrosstalkSources_1031719_32.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3c04595b-bffe-4c03-9a37-222a7f2aa7bd_isrForCrosstalkSources_1031719_32.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9965ea5f5eca958aefa9a81e6ea62587 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5a72814e-269d-4009-8958-42c6a09306e1_isrForCrosstalkSources_1031720_44.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5a72814e-269d-4009-8958-42c6a09306e1_isrForCrosstalkSources_1031720_44.stderr parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c6e8dfab812cfaf8a68a71bfe7f6070 with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/638f50df-48bb-4935-9535-e007459f7d7e_isrForCrosstalkSources_1031732_48.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/638f50df-48bb-4935-9535-e007459f7d7e_isrForCrosstalkSources_1031732_48.stderr parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5a8b52f5aa2e780b1db219308127c32 with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/665cca77-ec3b-432f-9678-5d9ea80dadaa_isrForCrosstalkSources_1031705_29.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/665cca77-ec3b-432f-9678-5d9ea80dadaa_isrForCrosstalkSources_1031705_29.stderr parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 621a24eccf90f1d3a073a0a6051edda8 with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/98fb536a-8880-44e8-ab58-b7e914f01fc6_isrForCrosstalkSources_1031716_11.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/98fb536a-8880-44e8-ab58-b7e914f01fc6_isrForCrosstalkSources_1031716_11.stderr parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09204e6107a5c46a3f1e9d70d198744e with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/12eac9b1-b3e8-4658-b0ce-6f5eff36b476_isrForCrosstalkSources_1031723_42.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/12eac9b1-b3e8-4658-b0ce-6f5eff36b476_isrForCrosstalkSources_1031723_42.stderr parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dbf7673655c32a1322ea0ed3b689986 with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8d76f3e4-4b2c-4588-ae24-bd425351bac1_isrForCrosstalkSources_1031707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8d76f3e4-4b2c-4588-ae24-bd425351bac1_isrForCrosstalkSources_1031707_2.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62fbeaaf32e52949a846349dcc1d8f6e with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6249170e-b387-4b79-aa5a-88e614450a40_isrForCrosstalkSources_1031728_5.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6249170e-b387-4b79-aa5a-88e614450a40_isrForCrosstalkSources_1031728_5.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79910d843b311649cebc16d4accdced8 with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9271b7bd-9f4f-41c3-9415-b1b97da5598b_isrForCrosstalkSources_1031729_31.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9271b7bd-9f4f-41c3-9415-b1b97da5598b_isrForCrosstalkSources_1031729_31.stderr parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f21b0ad8909110d5b5a03b6879bf2b6 with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/f4202191-185d-42d8-b034-27a5376eef15_isrForCrosstalkSources_1031707_17.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/f4202191-185d-42d8-b034-27a5376eef15_isrForCrosstalkSources_1031707_17.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 090ada0bc44fe56be50a5e0a7ecd38c9 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/6990de19-5752-4a41-ad41-096a7bd5d5a1_isrForCrosstalkSources_1031719_46.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/6990de19-5752-4a41-ad41-096a7bd5d5a1_isrForCrosstalkSources_1031719_46.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc6049d818dc7135a5ee2cbe320b501c with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f3cdf290-e1d6-4bd3-971f-ea7fc592c814_isrForCrosstalkSources_1031709_2.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f3cdf290-e1d6-4bd3-971f-ea7fc592c814_isrForCrosstalkSources_1031709_2.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a867ba007130cc1fffc6d2e06c2aa511 with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d3b960f2-a895-481d-b87a-46eafb3cc69f_isrForCrosstalkSources_1031731_19.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d3b960f2-a895-481d-b87a-46eafb3cc69f_isrForCrosstalkSources_1031731_19.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a44544ce45b7610a6eff34ce9f7f456 with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/077787f9-c7b7-4b55-bf99-75eddb59cb66_isrForCrosstalkSources_1031715_2.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/077787f9-c7b7-4b55-bf99-75eddb59cb66_isrForCrosstalkSources_1031715_2.stderr parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f904d0853bbca93030ceaddd13150f0 with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/20cd5d15-7d36-4ee3-b031-421527b150c5_isrForCrosstalkSources_1031700_39.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/20cd5d15-7d36-4ee3-b031-421527b150c5_isrForCrosstalkSources_1031700_39.stderr parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 739fe30a38eef4f2f6e69702f9c567a8 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a48a9dc0-e4d7-4c36-8ba1-14076e9565ff_isrForCrosstalkSources_1031709_50.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a48a9dc0-e4d7-4c36-8ba1-14076e9565ff_isrForCrosstalkSources_1031709_50.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e5e5649d3d4f9c2ac717ede8a177af0 with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2a0f0cfa-fc7d-485b-875e-fd2228a144e0_isrForCrosstalkSources_1031726_31.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2a0f0cfa-fc7d-485b-875e-fd2228a144e0_isrForCrosstalkSources_1031726_31.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f242e46c385b3df50446ef7738ca068 with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/e3d884da-9dcc-4c8e-bffb-6ae24af88341_isrForCrosstalkSources_1031715_9.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/e3d884da-9dcc-4c8e-bffb-6ae24af88341_isrForCrosstalkSources_1031715_9.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbfa2a459a2ca2a69502c63173bebc68 with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/bfd17d9c-5498-4faa-af1f-1c7de738e9ad_isrForCrosstalkSources_1031737_30.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/bfd17d9c-5498-4faa-af1f-1c7de738e9ad_isrForCrosstalkSources_1031737_30.stderr parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39fe4481268a0be7db73c823b69d7623 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8a00e4bb-f258-4f3b-96ee-7703fd088c1e_isrForCrosstalkSources_1031704_47.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8a00e4bb-f258-4f3b-96ee-7703fd088c1e_isrForCrosstalkSources_1031704_47.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67097b487acb58b5af7c5c7b3a2df117 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0a70c4ca-3a9b-45df-a343-093e9131d91a_isrForCrosstalkSources_1031705_22.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0a70c4ca-3a9b-45df-a343-093e9131d91a_isrForCrosstalkSources_1031705_22.stderr 2024-03-17 04:05:15 proc_lsst.multi:170 [INFO] [multi] job_id 17148044 proc_lsst.multi INFO: [multi] job_id 17148044 2024-03-17 04:05:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-17 04:05:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148044 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148044 parsl.executors.status_handling DEBUG: Launched block 99 on executor multi with job ID 17148044 parsl.executors.status_handling INFO: Allocated block ID 100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-100 2024-03-17 04:05:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 2024-03-17 04:05:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-17 04:05:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79c57a9b9a7e255471c871ec1a834c60 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0172a12b-84e8-4f47-b4a3-4938ca4c2d0c_isrForCrosstalkSources_1031711_2.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0172a12b-84e8-4f47-b4a3-4938ca4c2d0c_isrForCrosstalkSources_1031711_2.stderr parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea059e5bc28b95892f925a6876606f1e with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6ea3ee8c-c9b1-424b-96e9-39fa02f8153d_isrForCrosstalkSources_1031728_26.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6ea3ee8c-c9b1-424b-96e9-39fa02f8153d_isrForCrosstalkSources_1031728_26.stderr parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41256f712571698ecb905a38403ae55a with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/eeef0511-df72-4060-a06a-224f05dd244b_isrForCrosstalkSources_1031716_3.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/eeef0511-df72-4060-a06a-224f05dd244b_isrForCrosstalkSources_1031716_3.stderr parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fadc2a715fd65ee409d2b83b36c3bf5c with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ccd86a5a-19b5-4b40-ae28-d7ad56f86da6_isrForCrosstalkSources_1031725_18.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ccd86a5a-19b5-4b40-ae28-d7ad56f86da6_isrForCrosstalkSources_1031725_18.stderr parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db07f944a41279c9ad578217a983b638 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/6da9d678-0922-4aff-80ef-5c9188f9cf19_isrForCrosstalkSources_1031703_28.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/6da9d678-0922-4aff-80ef-5c9188f9cf19_isrForCrosstalkSources_1031703_28.stderr parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa96db6639854301f866de816f226029 with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/89359499-62ca-48ba-af00-89580dac413e_isrForCrosstalkSources_1031730_38.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/89359499-62ca-48ba-af00-89580dac413e_isrForCrosstalkSources_1031730_38.stderr parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74d9c59f6f8f9bd7210edfaa7c0b52be with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ae7b22ec-baac-46e4-9667-351857230fc8_isrForCrosstalkSources_1031737_49.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ae7b22ec-baac-46e4-9667-351857230fc8_isrForCrosstalkSources_1031737_49.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfc017b7b004cccada5929e3ae1ad122 with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/bd6615e7-6b46-429e-b801-304d1c8ad270_isrForCrosstalkSources_1031700_40.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/bd6615e7-6b46-429e-b801-304d1c8ad270_isrForCrosstalkSources_1031700_40.stderr 2024-03-17 04:05:16 proc_lsst.multi:170 [INFO] [multi] job_id 17148045 proc_lsst.multi INFO: [multi] job_id 17148045 2024-03-17 04:05:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-17 04:05:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148045 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148045 parsl.executors.status_handling DEBUG: Launched block 100 on executor multi with job ID 17148045 parsl.executors.status_handling INFO: Allocated block ID 101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-101 2024-03-17 04:05:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 2024-03-17 04:05:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-17 04:05:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ae2c6942afb4972ff45a2b100b3e820 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1c55d933-c7a1-4826-ba91-398ab361666d_isrForCrosstalkSources_1031706_20.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1c55d933-c7a1-4826-ba91-398ab361666d_isrForCrosstalkSources_1031706_20.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e0f04b6da4e4302592f09453100b7bd with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d196ef69-f6fa-40e3-9f45-236e8cc5c0f1_isrForCrosstalkSources_1031728_10.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d196ef69-f6fa-40e3-9f45-236e8cc5c0f1_isrForCrosstalkSources_1031728_10.stderr 2024-03-17 04:05:17 proc_lsst.multi:170 [INFO] [multi] job_id 17148046 proc_lsst.multi INFO: [multi] job_id 17148046 2024-03-17 04:05:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-17 04:05:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148046 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148046 parsl.executors.status_handling DEBUG: Launched block 101 on executor multi with job ID 17148046 parsl.executors.status_handling INFO: Allocated block ID 102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-102 2024-03-17 04:05:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 2024-03-17 04:05:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-17 04:05:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e57aea93fe3ce96aa911b23a3211436 with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/3ebc3330-4efc-427d-9bb5-c0caf72d333b_isrForCrosstalkSources_1031727_52.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/3ebc3330-4efc-427d-9bb5-c0caf72d333b_isrForCrosstalkSources_1031727_52.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d0473079c3c780344199527e89c3529 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/dd90c357-9ac7-450b-917a-e4e09ca97d2f_isrForCrosstalkSources_1031708_15.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/dd90c357-9ac7-450b-917a-e4e09ca97d2f_isrForCrosstalkSources_1031708_15.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0520a2cccecf209beb2b3a14d9b4961a with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c10f3430-99c6-46c0-86b6-af6e9f7eafa4_isrForCrosstalkSources_1031734_39.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c10f3430-99c6-46c0-86b6-af6e9f7eafa4_isrForCrosstalkSources_1031734_39.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d2fbd9fa11bdc5ae035452faa96585d with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e90ae8d1-c8b1-4679-a1ad-d28d057f91c2_isrForCrosstalkSources_1031712_29.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e90ae8d1-c8b1-4679-a1ad-d28d057f91c2_isrForCrosstalkSources_1031712_29.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40693cab575329036ea5424fa88c44a2 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/79ab02c6-d619-4eea-a5e1-9097a83e6e60_isrForCrosstalkSources_1031734_2.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/79ab02c6-d619-4eea-a5e1-9097a83e6e60_isrForCrosstalkSources_1031734_2.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60fa3b631b2e8bb0b9b14a6bf8687529 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/f8ea7daa-1e7f-4e7a-90b6-5462429691c1_isrForCrosstalkSources_1031725_22.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/f8ea7daa-1e7f-4e7a-90b6-5462429691c1_isrForCrosstalkSources_1031725_22.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 264b66f461a67734b3596b34194b064e with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/3e915396-68d7-417e-99a5-c9269afc39cd_isrForCrosstalkSources_1031724_46.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/3e915396-68d7-417e-99a5-c9269afc39cd_isrForCrosstalkSources_1031724_46.stderr parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 589a0447413e1d11d802a0c30b2ebe12 with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/08b4e6db-1894-493f-88a2-9a16ef927bc5_isrForCrosstalkSources_1031730_6.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/08b4e6db-1894-493f-88a2-9a16ef927bc5_isrForCrosstalkSources_1031730_6.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bfcb62f0d00126d6b2e11e7de155451 with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/e1190e8a-21ea-4b31-b333-a0f1ea8b6d83_isrForCrosstalkSources_1031711_52.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/e1190e8a-21ea-4b31-b333-a0f1ea8b6d83_isrForCrosstalkSources_1031711_52.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c8f2338956fe6940fcccef65fb2fb40 with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67a3aa5e-51e9-4b4e-ac0c-9d0fec4bcb38_isrForCrosstalkSources_1031728_57.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67a3aa5e-51e9-4b4e-ac0c-9d0fec4bcb38_isrForCrosstalkSources_1031728_57.stderr parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0771fb3c5630b8459b6fd777f8f74ce with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/43da2c44-de65-481a-871b-f65fcd39a62b_isrForCrosstalkSources_1031736_54.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/43da2c44-de65-481a-871b-f65fcd39a62b_isrForCrosstalkSources_1031736_54.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e4c58906fe42fa42b9f790234289672 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ab217f0-9b21-4f87-a76c-835dfd6a4bb0_isrForCrosstalkSources_1031719_49.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ab217f0-9b21-4f87-a76c-835dfd6a4bb0_isrForCrosstalkSources_1031719_49.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4406712d49fa6cc4a16dc08cc47473a1 with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/6ba78d59-5298-4eb8-aa6e-9e5859bd112b_isrForCrosstalkSources_1031712_44.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/6ba78d59-5298-4eb8-aa6e-9e5859bd112b_isrForCrosstalkSources_1031712_44.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b365449e9489ed6346fc72b2b832bb45 with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9d42aee9-585c-4fe6-bf87-14a92a1727ef_isrForCrosstalkSources_1031733_37.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9d42aee9-585c-4fe6-bf87-14a92a1727ef_isrForCrosstalkSources_1031733_37.stderr 2024-03-17 04:05:18 proc_lsst.multi:170 [INFO] [multi] job_id 17148047 proc_lsst.multi INFO: [multi] job_id 17148047 2024-03-17 04:05:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-17 04:05:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148047 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148047 parsl.executors.status_handling DEBUG: Launched block 102 on executor multi with job ID 17148047 parsl.executors.status_handling INFO: Allocated block ID 103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-103 2024-03-17 04:05:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 2024-03-17 04:05:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-17 04:05:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03e52b1f873a536ebce17cf2ad717bd1 with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3505b4c5-bd5e-433b-b8e8-b1372e8ace16_isrForCrosstalkSources_1031738_11.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3505b4c5-bd5e-433b-b8e8-b1372e8ace16_isrForCrosstalkSources_1031738_11.stderr 2024-03-17 04:05:19 proc_lsst.multi:170 [INFO] [multi] job_id 17148048 proc_lsst.multi INFO: [multi] job_id 17148048 2024-03-17 04:05:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-17 04:05:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148048 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148048 parsl.executors.status_handling DEBUG: Launched block 103 on executor multi with job ID 17148048 parsl.executors.status_handling INFO: Allocated block ID 104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-104 2024-03-17 04:05:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 2024-03-17 04:05:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-17 04:05:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fa5849148ae49d79982a53418c8ac97 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/98878813-26c7-4941-98bf-919150677cee_isrForCrosstalkSources_1031725_47.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/98878813-26c7-4941-98bf-919150677cee_isrForCrosstalkSources_1031725_47.stderr parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c9bb89d9af9e6863b12f85d89872672 with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a9df5cb2-bd44-4dca-8fed-30e8aaa3589b_isrForCrosstalkSources_1031736_62.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a9df5cb2-bd44-4dca-8fed-30e8aaa3589b_isrForCrosstalkSources_1031736_62.stderr 2024-03-17 04:05:20 proc_lsst.multi:170 [INFO] [multi] job_id 17148049 proc_lsst.multi INFO: [multi] job_id 17148049 2024-03-17 04:05:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-17 04:05:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148049 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148049 parsl.executors.status_handling DEBUG: Launched block 104 on executor multi with job ID 17148049 parsl.executors.status_handling INFO: Allocated block ID 105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-105 2024-03-17 04:05:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 2024-03-17 04:05:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-17 04:05:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c600952f0b9c2a5576a9c5d332bfdab6 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1021fb94-6e85-47a0-88d6-f400f7026b2c_isrForCrosstalkSources_1031735_25.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1021fb94-6e85-47a0-88d6-f400f7026b2c_isrForCrosstalkSources_1031735_25.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb0711a9b67d9c55dc2d3bb6b3f0d6c7 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/2c451775-3d48-4ca7-9436-76e262c661de_isrForCrosstalkSources_1031700_58.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/2c451775-3d48-4ca7-9436-76e262c661de_isrForCrosstalkSources_1031700_58.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 069bb50ed24d1f6c42ad1c1a7215c6e1 with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cec8b751-0217-4d2b-b126-23ad55bbfdfb_isrForCrosstalkSources_1031728_27.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cec8b751-0217-4d2b-b126-23ad55bbfdfb_isrForCrosstalkSources_1031728_27.stderr 2024-03-17 04:05:21 proc_lsst.multi:170 [INFO] [multi] job_id 17148050 proc_lsst.multi INFO: [multi] job_id 17148050 2024-03-17 04:05:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-17 04:05:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148050 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148050 parsl.executors.status_handling DEBUG: Launched block 105 on executor multi with job ID 17148050 parsl.executors.status_handling INFO: Allocated block ID 106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-106 2024-03-17 04:05:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 2024-03-17 04:05:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-17 04:05:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad24b79d9fb0d8228156461c1c00debf with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/963f76c1-f67c-4351-a161-87638ac92d80_isrForCrosstalkSources_1031699_39.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/963f76c1-f67c-4351-a161-87638ac92d80_isrForCrosstalkSources_1031699_39.stderr parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c849c50bb910cba3ca4ba70a956b569 with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/944d4a2a-d675-4134-ae58-3689a48a7e6d_isrForCrosstalkSources_1031724_19.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/944d4a2a-d675-4134-ae58-3689a48a7e6d_isrForCrosstalkSources_1031724_19.stderr parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b278220a1e3c343256426ccebee6d8bc with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/b3571dfc-e4f8-4eeb-8d57-89ce4090af92_isrForCrosstalkSources_1031731_55.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/b3571dfc-e4f8-4eeb-8d57-89ce4090af92_isrForCrosstalkSources_1031731_55.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3f17032815c47bac35b304aaeaaaf93 with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/1fd6e26e-6ae9-4af7-85cb-57b11a788ff7_isrForCrosstalkSources_1031717_8.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/1fd6e26e-6ae9-4af7-85cb-57b11a788ff7_isrForCrosstalkSources_1031717_8.stderr parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c173f12f8d5c428eb073820beceec05d with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/0f7c0dc8-f619-423f-bdae-57cf4ff3b5b5_isrForCrosstalkSources_1031729_44.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/0f7c0dc8-f619-423f-bdae-57cf4ff3b5b5_isrForCrosstalkSources_1031729_44.stderr parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7314e7d999647dc1c2409f5c482067e with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/6d1730f5-9f47-4ff1-afa8-dfd7056467f2_isrForCrosstalkSources_1031735_13.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/6d1730f5-9f47-4ff1-afa8-dfd7056467f2_isrForCrosstalkSources_1031735_13.stderr 2024-03-17 04:05:22 proc_lsst.multi:170 [INFO] [multi] job_id 17148051 proc_lsst.multi INFO: [multi] job_id 17148051 2024-03-17 04:05:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-17 04:05:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148051 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148051 parsl.executors.status_handling DEBUG: Launched block 106 on executor multi with job ID 17148051 parsl.executors.status_handling INFO: Allocated block ID 107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-107 2024-03-17 04:05:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 2024-03-17 04:05:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-17 04:05:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7827d8937a9aaea2b15a33e4976ce4c8 with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7fe46a34-796b-4fda-90ab-affe1b6aa8a3_isrForCrosstalkSources_1031702_40.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7fe46a34-796b-4fda-90ab-affe1b6aa8a3_isrForCrosstalkSources_1031702_40.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f0203ef2bc7430bf5919e39041d6fb3 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0faebf59-59ff-4936-b1b2-d4f5007a208a_isrForCrosstalkSources_1031707_14.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0faebf59-59ff-4936-b1b2-d4f5007a208a_isrForCrosstalkSources_1031707_14.stderr parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 536b6d0529c86157b14bd24a5a1559a1 with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/64bcec3a-a7b3-46a6-8c5a-f74b7202b522_isrForCrosstalkSources_1031713_27.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/64bcec3a-a7b3-46a6-8c5a-f74b7202b522_isrForCrosstalkSources_1031713_27.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c762a59c047b6797c55d8b4b23435000 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8bc60305-a80d-45d3-b1bd-4e75c60118ea_isrForCrosstalkSources_1031719_47.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8bc60305-a80d-45d3-b1bd-4e75c60118ea_isrForCrosstalkSources_1031719_47.stderr parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f29ffc29c37d8fa10abb856d64d57bd with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/78be94f4-0a28-466b-843b-04f69a4ab4b3_isrForCrosstalkSources_1031738_5.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/78be94f4-0a28-466b-843b-04f69a4ab4b3_isrForCrosstalkSources_1031738_5.stderr 2024-03-17 04:05:23 proc_lsst.multi:170 [INFO] [multi] job_id 17148052 proc_lsst.multi INFO: [multi] job_id 17148052 2024-03-17 04:05:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-17 04:05:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148052 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148052 parsl.executors.status_handling DEBUG: Launched block 107 on executor multi with job ID 17148052 parsl.executors.status_handling INFO: Allocated block ID 108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-108 2024-03-17 04:05:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 2024-03-17 04:05:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-17 04:05:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cc18ed6875f5b24531988380c65761a with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/01286372-aeb9-46ad-b153-f3052e7993a5_isrForCrosstalkSources_1031718_56.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/01286372-aeb9-46ad-b153-f3052e7993a5_isrForCrosstalkSources_1031718_56.stderr parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6f900cfdfbf307410d4b02661ad8b42 with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b37cf49e-364f-47bd-a73d-73d36f776c35_isrForCrosstalkSources_1031721_32.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b37cf49e-364f-47bd-a73d-73d36f776c35_isrForCrosstalkSources_1031721_32.stderr parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b70a279e3abef1c38dccebeb13d5325 with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/870fe78e-89e8-4b94-8598-94325cf78da8_isrForCrosstalkSources_1031710_33.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/870fe78e-89e8-4b94-8598-94325cf78da8_isrForCrosstalkSources_1031710_33.stderr parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dc27998e38a4418ef33cc663aa487f2 with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/1fe9b2b4-cfcf-4219-91f8-70a95276b99e_isrForCrosstalkSources_1031708_25.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/1fe9b2b4-cfcf-4219-91f8-70a95276b99e_isrForCrosstalkSources_1031708_25.stderr parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de5fd35da03411b33edd0128ed7f2922 with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4127eaf3-fbca-4e2d-b9aa-fb747799377a_isrForCrosstalkSources_1031723_28.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4127eaf3-fbca-4e2d-b9aa-fb747799377a_isrForCrosstalkSources_1031723_28.stderr parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 297173ef0eb1b61612191a6b1197b62d with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4d0c6663-19e1-4184-9cc1-11cafec47e1b_isrForCrosstalkSources_1031723_23.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4d0c6663-19e1-4184-9cc1-11cafec47e1b_isrForCrosstalkSources_1031723_23.stderr parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74ab5fcb63a0e58ae44aaa31262f8cb9 with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3cd4d129-e1ca-4847-8d9b-4f37cd5ec806_isrForCrosstalkSources_1031723_11.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3cd4d129-e1ca-4847-8d9b-4f37cd5ec806_isrForCrosstalkSources_1031723_11.stderr 2024-03-17 04:05:24 proc_lsst.multi:170 [INFO] [multi] job_id 17148053 proc_lsst.multi INFO: [multi] job_id 17148053 2024-03-17 04:05:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 53 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-17 04:05:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148053 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148053 parsl.executors.status_handling DEBUG: Launched block 108 on executor multi with job ID 17148053 parsl.executors.status_handling INFO: Allocated block ID 109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-109 2024-03-17 04:05:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 2024-03-17 04:05:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 53 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-17 04:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba1e8d54de4866657185ad322eb6881b with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/5c7c0c9b-c633-4059-a028-03d2777164b5_isrForCrosstalkSources_1031723_10.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/5c7c0c9b-c633-4059-a028-03d2777164b5_isrForCrosstalkSources_1031723_10.stderr parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c2dbed7f369f6fca303a5559c162eb6 with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6e695977-b440-4056-8f3e-31022cb366a6_isrForCrosstalkSources_1031723_19.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6e695977-b440-4056-8f3e-31022cb366a6_isrForCrosstalkSources_1031723_19.stderr parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b87541457daee73f0543f0c3a0719d15 with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b717c230-4150-4263-aa75-b3ff00f1f75e_isrForCrosstalkSources_1031707_11.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b717c230-4150-4263-aa75-b3ff00f1f75e_isrForCrosstalkSources_1031707_11.stderr parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a65cdf567ee04085a9bef97ea1a81e1 with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/1cbc9d8d-ff96-4344-95b7-3f834e08b017_isrForCrosstalkSources_1031702_41.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/1cbc9d8d-ff96-4344-95b7-3f834e08b017_isrForCrosstalkSources_1031702_41.stderr parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32bd33ac73f078e3041ec3342714803b with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/fad0b31a-50b3-4634-9616-1ee9819158b1_isrForCrosstalkSources_1031727_18.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/fad0b31a-50b3-4634-9616-1ee9819158b1_isrForCrosstalkSources_1031727_18.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d82526fa36e9f26237f708f5caba8942 with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2681fa5c-14a0-4ab6-91e0-9ecd3ce2d4d3_isrForCrosstalkSources_1031724_14.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2681fa5c-14a0-4ab6-91e0-9ecd3ce2d4d3_isrForCrosstalkSources_1031724_14.stderr 2024-03-17 04:05:24 proc_lsst.multi:170 [INFO] [multi] job_id 17148054 proc_lsst.multi INFO: [multi] job_id 17148054 2024-03-17 04:05:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 54 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 54 2024-03-17 04:05:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148054 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148054 parsl.executors.status_handling DEBUG: Launched block 109 on executor multi with job ID 17148054 parsl.executors.status_handling INFO: Allocated block ID 110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-110 2024-03-17 04:05:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 2024-03-17 04:05:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 54 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 54 2024-03-17 04:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a32bb1feaac4befa45b36b3c6d12b57 with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/f85df925-74ff-431f-a01b-21f591319150_isrForCrosstalkSources_1031738_29.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/f85df925-74ff-431f-a01b-21f591319150_isrForCrosstalkSources_1031738_29.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0079101c1af1b2f932ce64d5ee5231e9 with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/10ae0c34-60a6-42f6-9ae8-5707a25ff2c4_isrForCrosstalkSources_1031711_55.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/10ae0c34-60a6-42f6-9ae8-5707a25ff2c4_isrForCrosstalkSources_1031711_55.stderr parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4023d0f9b90e9d0528974032084293db with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/5dd52ba8-97e0-4bc1-9a7e-621bbfa00c6f_isrForCrosstalkSources_1031734_50.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/5dd52ba8-97e0-4bc1-9a7e-621bbfa00c6f_isrForCrosstalkSources_1031734_50.stderr parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e9220cbab89ef0e6674fc12f465d1b0 with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/cd9e0c78-5a95-44bc-b1c2-37d76d8ee46c_isrForCrosstalkSources_1031722_11.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/cd9e0c78-5a95-44bc-b1c2-37d76d8ee46c_isrForCrosstalkSources_1031722_11.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b3240e4181f938828ec6eb788033ae9 with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/211be384-be48-4516-9338-cffa6d07df85_isrForCrosstalkSources_1031702_38.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/211be384-be48-4516-9338-cffa6d07df85_isrForCrosstalkSources_1031702_38.stderr 2024-03-17 04:05:25 proc_lsst.multi:170 [INFO] [multi] job_id 17148055 proc_lsst.multi INFO: [multi] job_id 17148055 2024-03-17 04:05:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 55 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 55 2024-03-17 04:05:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148055 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148055 parsl.executors.status_handling DEBUG: Launched block 110 on executor multi with job ID 17148055 parsl.executors.status_handling INFO: Allocated block ID 111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-111 2024-03-17 04:05:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 2024-03-17 04:05:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 55 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 55 2024-03-17 04:05:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 404f79a94a07cb649c149d594baae04c with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/7cf51e6e-5c92-4aa5-a523-a1ec523f93f5_isrForCrosstalkSources_1031724_4.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/7cf51e6e-5c92-4aa5-a523-a1ec523f93f5_isrForCrosstalkSources_1031724_4.stderr parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94d4da9f98f2340aa251ddaa533ce47d with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d66c1a95-1e2d-48bd-a20a-2a3fba63a439_isrForCrosstalkSources_1031734_19.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d66c1a95-1e2d-48bd-a20a-2a3fba63a439_isrForCrosstalkSources_1031734_19.stderr 2024-03-17 04:05:25 proc_lsst.multi:170 [INFO] [multi] job_id 17148056 proc_lsst.multi INFO: [multi] job_id 17148056 2024-03-17 04:05:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 56 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 56 2024-03-17 04:05:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148056 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148056 parsl.executors.status_handling DEBUG: Launched block 111 on executor multi with job ID 17148056 parsl.executors.status_handling INFO: Allocated block ID 112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-112 2024-03-17 04:05:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 2024-03-17 04:05:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 56 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 56 2024-03-17 04:05:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39bef463347d5fe63ce414c6e91806fc with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3d415f8e-8809-40ca-99b1-f3b0c43482b7_isrForCrosstalkSources_1031716_8.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3d415f8e-8809-40ca-99b1-f3b0c43482b7_isrForCrosstalkSources_1031716_8.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1942d2a209d0bc7a358194d213877659 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/50c6b3c4-5084-4469-aaa6-2984846742db_isrForCrosstalkSources_1031700_3.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/50c6b3c4-5084-4469-aaa6-2984846742db_isrForCrosstalkSources_1031700_3.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af5df31ffd2e4b593342031e86a782bd with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/22818813-1eed-43ba-ab9b-d72edfd5086c_isrForCrosstalkSources_1031716_1.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/22818813-1eed-43ba-ab9b-d72edfd5086c_isrForCrosstalkSources_1031716_1.stderr parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2fc3194c1c34f413f856414c9cf7426 with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/74233290-0a52-4111-9c20-5b532134c1ff_isrForCrosstalkSources_1031720_5.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/74233290-0a52-4111-9c20-5b532134c1ff_isrForCrosstalkSources_1031720_5.stderr parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cfc9353121c7ee6356cf5dfda14c175 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/33b91735-e8bb-4ab4-b4ca-d2188f3d1089_isrForCrosstalkSources_1031729_19.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/33b91735-e8bb-4ab4-b4ca-d2188f3d1089_isrForCrosstalkSources_1031729_19.stderr parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 434fa0581b231fcc771704f175fd3f52 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a477c1f4-6fae-4876-a3e2-fe3717e74258_isrForCrosstalkSources_1031719_55.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a477c1f4-6fae-4876-a3e2-fe3717e74258_isrForCrosstalkSources_1031719_55.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1abf8794ebaa8ff3c7813c7ad08775 with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/3cece70b-72be-4fae-9776-3185cc8988bb_isrForCrosstalkSources_1031703_5.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/3cece70b-72be-4fae-9776-3185cc8988bb_isrForCrosstalkSources_1031703_5.stderr parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99babda8b7486bd8d9300a738309e90b with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/05d4e54b-dad1-4e6f-8ec8-2958bd6ff2c7_isrForCrosstalkSources_1031735_49.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/05d4e54b-dad1-4e6f-8ec8-2958bd6ff2c7_isrForCrosstalkSources_1031735_49.stderr parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa9acf3dad3753f9e3bf7273ba8ca240 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/4dba6235-aa47-4413-a7d7-cdf76114c59a_isrForCrosstalkSources_1031708_17.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/4dba6235-aa47-4413-a7d7-cdf76114c59a_isrForCrosstalkSources_1031708_17.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1bd76966bd6f94c19dc268c827bb51e with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/6a4c2764-4cbb-45d1-be13-7b5492401785_isrForCrosstalkSources_1031736_5.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/6a4c2764-4cbb-45d1-be13-7b5492401785_isrForCrosstalkSources_1031736_5.stderr parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed40d6890aeabaaad9fd50cbf685a2f3 with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/5c1eac5b-bff8-45e3-9c9e-f4d03bd7f8be_isrForCrosstalkSources_1031699_60.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/5c1eac5b-bff8-45e3-9c9e-f4d03bd7f8be_isrForCrosstalkSources_1031699_60.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 847fa2ccf0c68977216c89d54ac48b66 with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c94b1f63-01eb-463b-9475-4a763079f8be_isrForCrosstalkSources_1031725_28.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c94b1f63-01eb-463b-9475-4a763079f8be_isrForCrosstalkSources_1031725_28.stderr parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a641852a8c4df0a2c99f0270b4fd260f with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/cde3304f-4065-4bf5-9d98-b8907e394a90_isrForCrosstalkSources_1031713_58.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/cde3304f-4065-4bf5-9d98-b8907e394a90_isrForCrosstalkSources_1031713_58.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 286c5a322ebc7607fad3b6ded3ad9c6e with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/fdc6cae2-8f7c-4020-8094-b61ecd4266cc_isrForCrosstalkSources_1031699_35.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/fdc6cae2-8f7c-4020-8094-b61ecd4266cc_isrForCrosstalkSources_1031699_35.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7f869c23fab527bd4e360aa01cd22bb with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/436bacfe-65ae-427e-9d96-8c7f55153642_isrForCrosstalkSources_1031705_61.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/436bacfe-65ae-427e-9d96-8c7f55153642_isrForCrosstalkSources_1031705_61.stderr 2024-03-17 04:05:26 proc_lsst.multi:170 [INFO] [multi] job_id 17148057 proc_lsst.multi INFO: [multi] job_id 17148057 2024-03-17 04:05:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 57 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 57 2024-03-17 04:05:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148057 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148057 parsl.executors.status_handling DEBUG: Launched block 112 on executor multi with job ID 17148057 parsl.executors.status_handling INFO: Allocated block ID 113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-113 2024-03-17 04:05:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 2024-03-17 04:05:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 57 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 57 2024-03-17 04:05:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8912053711838f6b9e4ec38d21278c28 with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/4e97ebd5-0065-4c88-9e74-940a0dd02655_isrForCrosstalkSources_1031729_24.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/4e97ebd5-0065-4c88-9e74-940a0dd02655_isrForCrosstalkSources_1031729_24.stderr parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecce16df8fd3dd69c2a4e094e25fba3e with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/cfb61a35-db48-4413-b806-6656e0e6e8f1_isrForCrosstalkSources_1031734_40.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/cfb61a35-db48-4413-b806-6656e0e6e8f1_isrForCrosstalkSources_1031734_40.stderr parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ff1b0fce73b7c666ef54ece6c5f057b with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/fd6a19c7-6a73-40a4-b311-75ac36348439_isrForCrosstalkSources_1031730_56.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/fd6a19c7-6a73-40a4-b311-75ac36348439_isrForCrosstalkSources_1031730_56.stderr 2024-03-17 04:05:27 proc_lsst.multi:170 [INFO] [multi] job_id 17148058 proc_lsst.multi INFO: [multi] job_id 17148058 2024-03-17 04:05:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 58 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 58 2024-03-17 04:05:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148058 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148058 parsl.executors.status_handling DEBUG: Launched block 113 on executor multi with job ID 17148058 parsl.executors.status_handling INFO: Allocated block ID 114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-114 2024-03-17 04:05:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 2024-03-17 04:05:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 58 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 58 2024-03-17 04:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5e1abcf4bb28859b0fa4f8ba4a7430c with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/365f87a6-dae5-4349-90ef-9575bee7ad3c_isrForCrosstalkSources_1031727_38.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/365f87a6-dae5-4349-90ef-9575bee7ad3c_isrForCrosstalkSources_1031727_38.stderr parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afb2fe2bb4ee897b6d1cc4fd6f15b83d with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/bf8ded62-c9e9-452e-9b4a-9189e2f3809d_isrForCrosstalkSources_1031708_9.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/bf8ded62-c9e9-452e-9b4a-9189e2f3809d_isrForCrosstalkSources_1031708_9.stderr parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48228c66b02f9280b3356c491cc53d02 with result from task 830 parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5febd1be-d407-40a0-948d-d0b8a8242543_isrForCrosstalkSources_1031730_1.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5febd1be-d407-40a0-948d-d0b8a8242543_isrForCrosstalkSources_1031730_1.stderr 2024-03-17 04:05:27 proc_lsst.multi:170 [INFO] [multi] job_id 17148059 proc_lsst.multi INFO: [multi] job_id 17148059 2024-03-17 04:05:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 59 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 59 2024-03-17 04:05:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148059 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148059 parsl.executors.status_handling DEBUG: Launched block 114 on executor multi with job ID 17148059 parsl.executors.status_handling INFO: Allocated block ID 115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-115 2024-03-17 04:05:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 2024-03-17 04:05:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 59 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 59 2024-03-17 04:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization DEBUG: Storing app cache entry c446ad5f471d0e8a6750a1469b88e90e with result from task 773 parsl.dataflow.dflow DEBUG: Task 773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1e630d95-c07b-42a3-84f2-05b589305495_isrForCrosstalkSources_1031709_24.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1e630d95-c07b-42a3-84f2-05b589305495_isrForCrosstalkSources_1031709_24.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdf4db395045a0b5d86bbd36cefcdc41 with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cdf9e3a4-b640-4407-8b90-9f26f848e934_isrForCrosstalkSources_1031727_39.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cdf9e3a4-b640-4407-8b90-9f26f848e934_isrForCrosstalkSources_1031727_39.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e41b837b72fd8014e623a14ed7beb4ec with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/16cbf79c-b44d-442f-832e-96bd63f9a611_isrForCrosstalkSources_1031732_38.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/16cbf79c-b44d-442f-832e-96bd63f9a611_isrForCrosstalkSources_1031732_38.stderr parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c5a54f8d127a41ee2ac66e68cc9b32e with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/07d7b59a-bb37-4b33-900b-50479d45b131_isrForCrosstalkSources_1031731_26.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/07d7b59a-bb37-4b33-900b-50479d45b131_isrForCrosstalkSources_1031731_26.stderr parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f9a86662e3167baf44fda88b8982f7f with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e4dce8b9-b9c7-4fc4-afa5-c2947085ea81_isrForCrosstalkSources_1031716_39.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e4dce8b9-b9c7-4fc4-afa5-c2947085ea81_isrForCrosstalkSources_1031716_39.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4acdcb41ae8c4fb2e8d72ffcf7524fc3 with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/f2f9a5b4-c99b-477c-9271-ba7aa2764cac_isrForCrosstalkSources_1031711_25.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/f2f9a5b4-c99b-477c-9271-ba7aa2764cac_isrForCrosstalkSources_1031711_25.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fef62d5abd1229875d8300eb496d25e with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ab2ef105-5f7f-4233-ab67-d373077e56dc_isrForCrosstalkSources_1031715_26.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ab2ef105-5f7f-4233-ab67-d373077e56dc_isrForCrosstalkSources_1031715_26.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cef07808c85337565235b2e482839033 with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/185a99cd-630e-4b30-8a7f-f559a08db134_isrForCrosstalkSources_1031723_13.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/185a99cd-630e-4b30-8a7f-f559a08db134_isrForCrosstalkSources_1031723_13.stderr parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a382e226442a734488d7cc2ec524d39f with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/eac9e417-f421-4608-9d2d-faebf6cd9858_isrForCrosstalkSources_1031724_20.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/eac9e417-f421-4608-9d2d-faebf6cd9858_isrForCrosstalkSources_1031724_20.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 036371fa4e2a89f9cf971665390928b7 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4f8e1dd3-f12a-4869-b6d1-ec3806c9a590_isrForCrosstalkSources_1031732_14.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4f8e1dd3-f12a-4869-b6d1-ec3806c9a590_isrForCrosstalkSources_1031732_14.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02b535fd4e4d91c8c3175d9f12b09f32 with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f7a211e1-f383-4386-a182-d56623dfedf4_isrForCrosstalkSources_1031700_43.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f7a211e1-f383-4386-a182-d56623dfedf4_isrForCrosstalkSources_1031700_43.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3446a14b964f64ba81819481fb6658a8 with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3f7aaf5a-9e60-44bc-affe-edf01f2024c3_isrForCrosstalkSources_1031732_7.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3f7aaf5a-9e60-44bc-affe-edf01f2024c3_isrForCrosstalkSources_1031732_7.stderr parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47fd24d31c70d8788bdcde10e5644f35 with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5c0ae407-cc64-4db4-8692-52eddac4fbf2_isrForCrosstalkSources_1031703_23.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5c0ae407-cc64-4db4-8692-52eddac4fbf2_isrForCrosstalkSources_1031703_23.stderr parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 889ee4843348b7d2105930c4d44bb25c with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c88d44e9-8f0f-4526-a149-e90d670c8162_isrForCrosstalkSources_1031715_45.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c88d44e9-8f0f-4526-a149-e90d670c8162_isrForCrosstalkSources_1031715_45.stderr parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16a1a1348062442562be35ab8e6e1481 with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/bb37b8b1-a71d-4e3c-8586-99a9a6e7d885_isrForCrosstalkSources_1031712_1.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/bb37b8b1-a71d-4e3c-8586-99a9a6e7d885_isrForCrosstalkSources_1031712_1.stderr parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 186a9557b9403427e151b6504ba22a47 with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/71e5985d-e887-4d4a-bf56-6f9ad091ffcd_isrForCrosstalkSources_1031722_36.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/71e5985d-e887-4d4a-bf56-6f9ad091ffcd_isrForCrosstalkSources_1031722_36.stderr parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd3d78d016afda8b80b0b52b5fffb1d8 with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/da54ca92-c448-405e-b27e-f8035f7107cc_isrForCrosstalkSources_1031728_14.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/da54ca92-c448-405e-b27e-f8035f7107cc_isrForCrosstalkSources_1031728_14.stderr parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66e017fa564586d256df49c2e0474e4f with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/9f29e782-4085-4f24-9d41-28a65c3b311d_isrForCrosstalkSources_1031703_59.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/9f29e782-4085-4f24-9d41-28a65c3b311d_isrForCrosstalkSources_1031703_59.stderr parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2e89323eee2e449633a4e49a95a507b with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/1f452135-cf6f-4806-b980-ddc87d7fca8f_isrForCrosstalkSources_1031727_6.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/1f452135-cf6f-4806-b980-ddc87d7fca8f_isrForCrosstalkSources_1031727_6.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca8d70d9a241980215e3e030af19baac with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/58ac916b-2d28-49d9-ae9f-fe1c19b69b89_isrForCrosstalkSources_1031701_46.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/58ac916b-2d28-49d9-ae9f-fe1c19b69b89_isrForCrosstalkSources_1031701_46.stderr parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fedd1996d33d0bc069ad822ccedd5c9a with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f6bfb9ac-862e-410f-9fed-7909a3ba40bf_isrForCrosstalkSources_1031727_29.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f6bfb9ac-862e-410f-9fed-7909a3ba40bf_isrForCrosstalkSources_1031727_29.stderr 2024-03-17 04:05:30 proc_lsst.multi:170 [INFO] [multi] job_id 17148061 proc_lsst.multi INFO: [multi] job_id 17148061 2024-03-17 04:05:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 60 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 60 2024-03-17 04:05:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148061 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148061 parsl.executors.status_handling DEBUG: Launched block 115 on executor multi with job ID 17148061 parsl.executors.status_handling INFO: Allocated block ID 116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-116 2024-03-17 04:05:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 2024-03-17 04:05:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 60 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 60 2024-03-17 04:05:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90ca7276b5f4f0a8fd756a9f350f1c3e with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/270b19cf-c87e-4c92-9dee-e1c2cf8325c9_isrForCrosstalkSources_1031725_49.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/270b19cf-c87e-4c92-9dee-e1c2cf8325c9_isrForCrosstalkSources_1031725_49.stderr parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d2dea2d726b5e1c5ac5382965b98991 with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/16f4879c-16b2-4d41-966f-273e34ab3e8c_isrForCrosstalkSources_1031703_40.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/16f4879c-16b2-4d41-966f-273e34ab3e8c_isrForCrosstalkSources_1031703_40.stderr parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c629707624ba3e9090437a55f1fce2c1 with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/260547d1-38e6-4a94-b29f-c05b4a9d13f4_isrForCrosstalkSources_1031711_51.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/260547d1-38e6-4a94-b29f-c05b4a9d13f4_isrForCrosstalkSources_1031711_51.stderr parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63788038a1a7ef3fe1c765d6bf2ef722 with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/c2e8043b-ce64-401e-813e-2295ad07e98e_isrForCrosstalkSources_1031724_6.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/c2e8043b-ce64-401e-813e-2295ad07e98e_isrForCrosstalkSources_1031724_6.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50e59713264774282a2a4750394b0b4f with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/aa75606b-339d-41e6-aac8-bfd1d95fdb99_isrForCrosstalkSources_1031707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/aa75606b-339d-41e6-aac8-bfd1d95fdb99_isrForCrosstalkSources_1031707_31.stderr parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35d3197c08afd90d826f0f57c5365efa with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4d40d505-a680-4344-955f-25cb8c1c90ad_isrForCrosstalkSources_1031720_42.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4d40d505-a680-4344-955f-25cb8c1c90ad_isrForCrosstalkSources_1031720_42.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c9a7bc33d5cf214a6c41bc21d82b486 with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2fbde783-0e97-445a-b08c-ff74a01045ec_isrForCrosstalkSources_1031736_10.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2fbde783-0e97-445a-b08c-ff74a01045ec_isrForCrosstalkSources_1031736_10.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26b889119d8e7a88a3418aacea493b73 with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8e52f881-bd8b-46d9-a939-f8ece7dad40f_isrForCrosstalkSources_1031719_21.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8e52f881-bd8b-46d9-a939-f8ece7dad40f_isrForCrosstalkSources_1031719_21.stderr parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2407cbcc07f7b1d18cc1c5b953d0b99d with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c9a1320d-9f51-4e57-8dbd-2005398775ec_isrForCrosstalkSources_1031702_11.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c9a1320d-9f51-4e57-8dbd-2005398775ec_isrForCrosstalkSources_1031702_11.stderr parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07f5ceb1d58c7cb1cc7dbec4e90abe6f with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/012b02fd-acdc-46b3-8b1d-8c4eda7a340d_isrForCrosstalkSources_1031733_7.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/012b02fd-acdc-46b3-8b1d-8c4eda7a340d_isrForCrosstalkSources_1031733_7.stderr parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b471ac5ef765c51c4e0206c5a45c5ca with result from task 866 parsl.dataflow.dflow DEBUG: Task 866 checkpointed 2024-03-17 04:05:31 proc_lsst.multi:170 [INFO] [multi] job_id 17148062 proc_lsst.multi INFO: [multi] job_id 17148062 2024-03-17 04:05:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 61 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 61 2024-03-17 04:05:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148062 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148062 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.executors.status_handling DEBUG: Launched block 116 on executor multi with job ID 17148062 parsl.executors.status_handling INFO: Allocated block ID 117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-117 2024-03-17 04:05:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 2024-03-17 04:05:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 61 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 61 2024-03-17 04:05:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/7cba93b2-6e13-4728-8a16-5eab091cb189_isrForCrosstalkSources_1031728_44.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/7cba93b2-6e13-4728-8a16-5eab091cb189_isrForCrosstalkSources_1031728_44.stderr parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eb387e779161422499564d704e10d7c with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/da416623-4ab6-4928-9a1d-4deaebdf8189_isrForCrosstalkSources_1031725_15.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/da416623-4ab6-4928-9a1d-4deaebdf8189_isrForCrosstalkSources_1031725_15.stderr parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ac06d1500808e7c1b4a3e9641b20804 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/812de840-2341-49e9-ab78-bfb2470a59ae_isrForCrosstalkSources_1031737_58.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/812de840-2341-49e9-ab78-bfb2470a59ae_isrForCrosstalkSources_1031737_58.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff0969037b32046cef336b47858c4243 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/9b2da222-db3f-4cdd-92cb-55648dc421dd_isrForCrosstalkSources_1031720_16.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/9b2da222-db3f-4cdd-92cb-55648dc421dd_isrForCrosstalkSources_1031720_16.stderr parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77ccb132cb2c4dde50080af2e8e00766 with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/30bd2e2d-f62f-4904-8044-c81bfa8902bc_isrForCrosstalkSources_1031719_17.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/30bd2e2d-f62f-4904-8044-c81bfa8902bc_isrForCrosstalkSources_1031719_17.stderr parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa928f6da027b9719ab589713c5ab5d8 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1fd2e7af-6228-4316-85bd-641d0ad2b23c_isrForCrosstalkSources_1031711_11.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1fd2e7af-6228-4316-85bd-641d0ad2b23c_isrForCrosstalkSources_1031711_11.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79b02de3483eb06089e899d1e0d05e9a with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/24054c14-51e0-4520-b74f-d3d3d0c9c1c9_isrForCrosstalkSources_1031722_9.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/24054c14-51e0-4520-b74f-d3d3d0c9c1c9_isrForCrosstalkSources_1031722_9.stderr parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 799d925a1f72fc2681200fd5ba07f8a5 with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/c81dd6a0-fa25-4a91-b192-fc80219b9640_isrForCrosstalkSources_1031720_39.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/c81dd6a0-fa25-4a91-b192-fc80219b9640_isrForCrosstalkSources_1031720_39.stderr parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f677d7599e6e43d991bff4f2cea55ce with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/7fe3d535-9a52-4813-b806-20ae06bc8a40_isrForCrosstalkSources_1031720_8.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/7fe3d535-9a52-4813-b806-20ae06bc8a40_isrForCrosstalkSources_1031720_8.stderr parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4454160524e616e02b15fb989042cdba with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/2488e2f2-f445-43b1-b29f-8b78a3f86c0b_isrForCrosstalkSources_1031732_5.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/2488e2f2-f445-43b1-b29f-8b78a3f86c0b_isrForCrosstalkSources_1031732_5.stderr parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 121c3dc2d094aa3d5708d5341b8590b0 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c831a806-5b9f-40e6-82c2-c45dcbde818d_isrForCrosstalkSources_1031738_55.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c831a806-5b9f-40e6-82c2-c45dcbde818d_isrForCrosstalkSources_1031738_55.stderr parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f5fba04361195e4513c3b6fd33290e2 with result from task 772 parsl.dataflow.dflow DEBUG: Task 772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/76ba6ff4-42bc-4d74-886e-30fa92b83df0_isrForCrosstalkSources_1031709_37.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/76ba6ff4-42bc-4d74-886e-30fa92b83df0_isrForCrosstalkSources_1031709_37.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f6c59798e8413ec6b941d172b53c985 with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/db182699-a23c-4d41-a52f-f7a014053f6b_isrForCrosstalkSources_1031733_43.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/db182699-a23c-4d41-a52f-f7a014053f6b_isrForCrosstalkSources_1031733_43.stderr parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91ea41028dea304a79dee393ece23d65 with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ea614c9a-2363-4784-929d-d2a18c8d2ec6_isrForCrosstalkSources_1031709_26.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ea614c9a-2363-4784-929d-d2a18c8d2ec6_isrForCrosstalkSources_1031709_26.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc98c5a355151b9ab125aeb98daf94cb with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/16a9ddf9-991f-4d14-986c-5bcc094fa110_isrForCrosstalkSources_1031712_11.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/16a9ddf9-991f-4d14-986c-5bcc094fa110_isrForCrosstalkSources_1031712_11.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d833e6fcd18e2520ef3e331e3870696c with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/85dabb50-5845-4ec0-9e19-e426623a119e_isrForCrosstalkSources_1031702_57.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/85dabb50-5845-4ec0-9e19-e426623a119e_isrForCrosstalkSources_1031702_57.stderr parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c9907c2bad3b00ca9d06042012df0e1 with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3659335a-5e14-4db9-8474-af74c9317d6b_isrForCrosstalkSources_1031738_12.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3659335a-5e14-4db9-8474-af74c9317d6b_isrForCrosstalkSources_1031738_12.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fc8b3f2537ab96417c5a98412535984 with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/977b737c-4008-4d47-8ed6-a92685f4609c_isrForCrosstalkSources_1031721_40.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/977b737c-4008-4d47-8ed6-a92685f4609c_isrForCrosstalkSources_1031721_40.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f0b76919fa306f9057eb21c8d033305 with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c480af5b-a820-4e7d-928c-9555b71ab6c4_isrForCrosstalkSources_1031706_42.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c480af5b-a820-4e7d-928c-9555b71ab6c4_isrForCrosstalkSources_1031706_42.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c4903e2c9143e4ad2471953063093ed with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1814685b-0861-4df4-a127-b750e5a29845_isrForCrosstalkSources_1031700_50.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1814685b-0861-4df4-a127-b750e5a29845_isrForCrosstalkSources_1031700_50.stderr parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9771ba77408fae031f422883d00f5f4f with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/b51b2cb0-a979-47cd-a782-7905b3c38ca3_isrForCrosstalkSources_1031727_31.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/b51b2cb0-a979-47cd-a782-7905b3c38ca3_isrForCrosstalkSources_1031727_31.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2983ec95ba08df01340b8a20c300542 with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/be5ee021-ae19-4c1d-a59e-ed1ca99ba651_isrForCrosstalkSources_1031733_4.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/be5ee021-ae19-4c1d-a59e-ed1ca99ba651_isrForCrosstalkSources_1031733_4.stderr parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c4bdd482e8de67ba0860d026cb8adf4 with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c2ea52d6-231e-446b-8c51-54b7d61883ea_isrForCrosstalkSources_1031712_45.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c2ea52d6-231e-446b-8c51-54b7d61883ea_isrForCrosstalkSources_1031712_45.stderr parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7594b55299bb836e4be2d71bea433f4c with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/49140586-e353-4e42-a77e-d54eba0436b9_isrForCrosstalkSources_1031709_41.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/49140586-e353-4e42-a77e-d54eba0436b9_isrForCrosstalkSources_1031709_41.stderr parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1807484cb8acd8c42f9781b3864ce021 with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/73af8567-f9ae-401c-a115-9ad796e60674_isrForCrosstalkSources_1031717_36.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/73af8567-f9ae-401c-a115-9ad796e60674_isrForCrosstalkSources_1031717_36.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2b52028d0a798bc89b5b5f1e9c0a724 with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/c745b8bd-0a54-4d9e-96eb-ce814b035a41_isrForCrosstalkSources_1031713_51.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/c745b8bd-0a54-4d9e-96eb-ce814b035a41_isrForCrosstalkSources_1031713_51.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a96f904843978866457f6a62c898c5e with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8d0806e8-ed61-4c31-8151-6f18ad51e45b_isrForCrosstalkSources_1031730_10.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8d0806e8-ed61-4c31-8151-6f18ad51e45b_isrForCrosstalkSources_1031730_10.stderr parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb912293cb9204d005027cf14af4288d with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c0278e48-e9ce-4945-8599-107c91373ba7_isrForCrosstalkSources_1031710_20.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c0278e48-e9ce-4945-8599-107c91373ba7_isrForCrosstalkSources_1031710_20.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 027e8ff65dafe1d51f73be119eeefb0f with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/5e09c646-f250-47ac-92cd-357e49748867_isrForCrosstalkSources_1031708_8.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/5e09c646-f250-47ac-92cd-357e49748867_isrForCrosstalkSources_1031708_8.stderr parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 887a264816949b0ea682fc4f4275e008 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/b1205a50-85b2-4669-bb8d-dc255b577277_isrForCrosstalkSources_1031734_37.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/b1205a50-85b2-4669-bb8d-dc255b577277_isrForCrosstalkSources_1031734_37.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2518a48be200879c9a3cfa53a9d04476 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/74de8857-7d00-4d13-9080-d16d07c81ea9_isrForCrosstalkSources_1031712_15.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/74de8857-7d00-4d13-9080-d16d07c81ea9_isrForCrosstalkSources_1031712_15.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86776c9eea0d08f970d7f11eb2cf7a10 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7b2f3f29-2b4b-4ef2-94bb-57800e76afb4_isrForCrosstalkSources_1031721_39.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/7b2f3f29-2b4b-4ef2-94bb-57800e76afb4_isrForCrosstalkSources_1031721_39.stderr parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbcc6df33a92b3a055e9b6d14222a352 with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4deed72c-ba8a-49cc-b345-8d5143bd78e3_isrForCrosstalkSources_1031718_32.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4deed72c-ba8a-49cc-b345-8d5143bd78e3_isrForCrosstalkSources_1031718_32.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aacc89116334bac733e871fa643db1c5 with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f041b2fc-2ea5-400d-a5ac-386d45cfa6af_isrForCrosstalkSources_1031716_42.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f041b2fc-2ea5-400d-a5ac-386d45cfa6af_isrForCrosstalkSources_1031716_42.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ff0ab9935899c3151895fe0cc242ed6 with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b5eeae9e-81bd-4088-b568-295019b602f4_isrForCrosstalkSources_1031718_2.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b5eeae9e-81bd-4088-b568-295019b602f4_isrForCrosstalkSources_1031718_2.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58f172edcbf7ff5dc816dce46c25d08a with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/24a6ccd2-f1ff-4846-bdec-b37d187ea54b_isrForCrosstalkSources_1031726_44.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/24a6ccd2-f1ff-4846-bdec-b37d187ea54b_isrForCrosstalkSources_1031726_44.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a9b25d89055696c05ce894946b4ac95 with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/aa9dd174-6339-4e26-8060-0b45d487b614_isrForCrosstalkSources_1031729_8.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/aa9dd174-6339-4e26-8060-0b45d487b614_isrForCrosstalkSources_1031729_8.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95b22a6f18c52ac23b2c4c204be4db38 with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c0d6bcec-eea3-403d-a429-e9e3c83fc3dd_isrForCrosstalkSources_1031703_3.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c0d6bcec-eea3-403d-a429-e9e3c83fc3dd_isrForCrosstalkSources_1031703_3.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 851005decd5384da57216630d4a883c2 with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/0eac1ee7-ce95-4d56-8273-8895a9f49deb_isrForCrosstalkSources_1031709_52.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/0eac1ee7-ce95-4d56-8273-8895a9f49deb_isrForCrosstalkSources_1031709_52.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03a07187b59bc0d5deaccadf25648305 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ccccfe57-0341-4b68-a436-bd4a1442ca68_isrForCrosstalkSources_1031726_15.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ccccfe57-0341-4b68-a436-bd4a1442ca68_isrForCrosstalkSources_1031726_15.stderr parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b2fce76d94c7f73860dbd5f7ec76d7a with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/de2881a7-bd1d-4b7d-957a-774fb4c049d6_isrForCrosstalkSources_1031704_12.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/de2881a7-bd1d-4b7d-957a-774fb4c049d6_isrForCrosstalkSources_1031704_12.stderr parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry deeb876a6fc146925276f3d514c56bb0 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0722407d-bd59-46ec-84eb-15598f018e7d_isrForCrosstalkSources_1031714_14.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0722407d-bd59-46ec-84eb-15598f018e7d_isrForCrosstalkSources_1031714_14.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77b7d8f7a16d12f50564662bd052090b with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/9963c37c-5a3b-4dd9-9070-b3e83dbab6ee_isrForCrosstalkSources_1031736_15.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/9963c37c-5a3b-4dd9-9070-b3e83dbab6ee_isrForCrosstalkSources_1031736_15.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e64ea1a119b48c22251ac0e86b6eb9f5 with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/af12f0e1-ba0a-4726-8c64-620356812c82_isrForCrosstalkSources_1031737_37.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/af12f0e1-ba0a-4726-8c64-620356812c82_isrForCrosstalkSources_1031737_37.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b235b73c75a03fc590f19febe4659f23 with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/926eb105-5f4c-45f2-a511-a97e836ad649_isrForCrosstalkSources_1031699_36.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/926eb105-5f4c-45f2-a511-a97e836ad649_isrForCrosstalkSources_1031699_36.stderr parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fd15dfd73093399f5ee93897bcf4b33 with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cfe8e82b-dd7b-4a3e-9f13-d72d694513da_isrForCrosstalkSources_1031716_35.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cfe8e82b-dd7b-4a3e-9f13-d72d694513da_isrForCrosstalkSources_1031716_35.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53f9a2c5e378e836c7f7c449f92b9457 with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/42ea1df2-0399-497e-84aa-bd7ee80c1b6a_isrForCrosstalkSources_1031727_16.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/42ea1df2-0399-497e-84aa-bd7ee80c1b6a_isrForCrosstalkSources_1031727_16.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff8234a61af732fce0d327ab8955c729 with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3a5e41bb-5e6b-493b-8636-3017ab319d80_isrForCrosstalkSources_1031736_24.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3a5e41bb-5e6b-493b-8636-3017ab319d80_isrForCrosstalkSources_1031736_24.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96d0bf70f7edb85e8baf5e7c86186873 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6e95fc96-2cba-4660-9045-08905af852dc_isrForCrosstalkSources_1031728_38.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/6e95fc96-2cba-4660-9045-08905af852dc_isrForCrosstalkSources_1031728_38.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d0ba444df4a6072f25ce888104d24f8 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/90eb9ac1-853d-4a9a-9193-6a66201c50f0_isrForCrosstalkSources_1031706_9.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/90eb9ac1-853d-4a9a-9193-6a66201c50f0_isrForCrosstalkSources_1031706_9.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85f1e3f8bd51639337bffff2fe580267 with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/cf01b3af-dcb4-4893-8ece-cd9e242302a6_isrForCrosstalkSources_1031719_59.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/cf01b3af-dcb4-4893-8ece-cd9e242302a6_isrForCrosstalkSources_1031719_59.stderr parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb6d01b21283f10ad9a3165c0ff01ec3 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ac093d7-a4bc-4f3e-b5a5-ec55aec84d80_isrForCrosstalkSources_1031719_43.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ac093d7-a4bc-4f3e-b5a5-ec55aec84d80_isrForCrosstalkSources_1031719_43.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a489cfb432f16167477694d1537bec7 with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a8998a46-dbac-4769-ad4f-18dc9be86c10_isrForCrosstalkSources_1031726_36.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a8998a46-dbac-4769-ad4f-18dc9be86c10_isrForCrosstalkSources_1031726_36.stderr parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25c357e705d452a2b06161126f9389a3 with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/50cae9c3-d7bb-4e2b-8e8c-6879f108397b_isrForCrosstalkSources_1031732_31.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/50cae9c3-d7bb-4e2b-8e8c-6879f108397b_isrForCrosstalkSources_1031732_31.stderr parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4129652d6d9bda6b28d0039d4b455bb with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/5cfacb1e-c223-4046-8fb1-8375d6eb63b6_isrForCrosstalkSources_1031738_50.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/5cfacb1e-c223-4046-8fb1-8375d6eb63b6_isrForCrosstalkSources_1031738_50.stderr parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df2e46ceee01f9a651ec4151fdc95a6f with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/202dd593-41ad-4c7a-88a8-3001b7018776_isrForCrosstalkSources_1031711_44.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/202dd593-41ad-4c7a-88a8-3001b7018776_isrForCrosstalkSources_1031711_44.stderr parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5150d04b5d77082084649cf7333c06da with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e072f816-d490-45f3-a6d0-17516b027af7_isrForCrosstalkSources_1031738_28.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e072f816-d490-45f3-a6d0-17516b027af7_isrForCrosstalkSources_1031738_28.stderr parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27411daa1007068257d466eff2897313 with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8a9be04f-f5b8-4f67-bd3b-d92a67326fd1_isrForCrosstalkSources_1031735_56.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8a9be04f-f5b8-4f67-bd3b-d92a67326fd1_isrForCrosstalkSources_1031735_56.stderr parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 900df9f6cfdb68d433a4bd82970baf8a with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/07fa0ffe-4db0-4a0c-9015-a1e155b67ae5_isrForCrosstalkSources_1031706_13.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/07fa0ffe-4db0-4a0c-9015-a1e155b67ae5_isrForCrosstalkSources_1031706_13.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ca72083ea9bac047b49680b5001f1e9 with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/cbd00b3b-93ec-486c-bc23-050819a4e274_isrForCrosstalkSources_1031731_36.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/cbd00b3b-93ec-486c-bc23-050819a4e274_isrForCrosstalkSources_1031731_36.stderr parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e1a567ffa74cceb37007ec5d8ea9339 with result from task 874 parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/abe80ace-181b-405e-a2fd-e057ee62a7df_isrForCrosstalkSources_1031726_1.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/abe80ace-181b-405e-a2fd-e057ee62a7df_isrForCrosstalkSources_1031726_1.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81a3a8d732e256998f06869fdebbbc49 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/5df39d20-69b1-4607-af02-7adb349ff84f_isrForCrosstalkSources_1031734_17.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/5df39d20-69b1-4607-af02-7adb349ff84f_isrForCrosstalkSources_1031734_17.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c7bbdfc8e9ccb7bdd36a3acb2fc217a with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cdc2bbe8-828e-4baf-8995-e42ef5ee29e3_isrForCrosstalkSources_1031716_50.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cdc2bbe8-828e-4baf-8995-e42ef5ee29e3_isrForCrosstalkSources_1031716_50.stderr parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be9615e9377b95baaa79b90563f92fe3 with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2f0b08dc-15c4-498c-a0f7-30ac4e781e08_isrForCrosstalkSources_1031709_54.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2f0b08dc-15c4-498c-a0f7-30ac4e781e08_isrForCrosstalkSources_1031709_54.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a17e684922df059319772f2e60e0aff with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/3cbfe7c4-c63f-449c-8df9-6a56314ac3b3_isrForCrosstalkSources_1031730_2.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/3cbfe7c4-c63f-449c-8df9-6a56314ac3b3_isrForCrosstalkSources_1031730_2.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b4178f04137c3ac22cda8f72433a8d4 with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/5a00d4f5-1b33-400e-ac23-1536997e086e_isrForCrosstalkSources_1031737_43.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/5a00d4f5-1b33-400e-ac23-1536997e086e_isrForCrosstalkSources_1031737_43.stderr parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c86a7ab17d2e738b7387236df57ba7fe with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f092f259-dbcb-4c4b-b4ca-0a083394e8a0_isrForCrosstalkSources_1031705_57.stdout 2024-03-17 04:05:34 proc_lsst.multi:170 [INFO] [multi] job_id 17148063 proc_lsst.multi INFO: [multi] job_id 17148063 2024-03-17 04:05:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 62 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 62 2024-03-17 04:05:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148063 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148063 parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f092f259-dbcb-4c4b-b4ca-0a083394e8a0_isrForCrosstalkSources_1031705_57.stderr parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 552b43acf457003e098ed406f5a4241f with result from task 902 parsl.executors.status_handling DEBUG: Launched block 117 on executor multi with job ID 17148063 parsl.executors.status_handling INFO: Allocated block ID 118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-118 2024-03-17 04:05:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 2024-03-17 04:05:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 62 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 62 2024-03-17 04:05:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/07261fda-ad71-4f72-bb63-4af523626fba_isrForCrosstalkSources_1031724_61.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/07261fda-ad71-4f72-bb63-4af523626fba_isrForCrosstalkSources_1031724_61.stderr proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7f0a6ab536474f13b533d2ae7b5c7fd with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/87daf0cc-dd80-4d2d-aef1-6ae47d2080b0_isrForCrosstalkSources_1031706_6.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/87daf0cc-dd80-4d2d-aef1-6ae47d2080b0_isrForCrosstalkSources_1031706_6.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dd46afb07be01ab3d5473f7103347d9 with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/9e71acac-ae25-4952-965d-e559a40d009d_isrForCrosstalkSources_1031702_51.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/9e71acac-ae25-4952-965d-e559a40d009d_isrForCrosstalkSources_1031702_51.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ad473c93a0ee94ed9fdaf42375f8ee1 with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b23fc4e9-11e2-4951-967e-5ae7856dfd82_isrForCrosstalkSources_1031707_43.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b23fc4e9-11e2-4951-967e-5ae7856dfd82_isrForCrosstalkSources_1031707_43.stderr parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13e21675d6654725717f906a5a71b91b with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/fce4cdb5-ddab-416a-a40c-60b8f86f7fc0_isrForCrosstalkSources_1031704_39.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/fce4cdb5-ddab-416a-a40c-60b8f86f7fc0_isrForCrosstalkSources_1031704_39.stderr parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b60bff48bfc154180a9e2304e9cbfc6b with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/dccc054c-81f7-4450-aeac-cec6d907a509_isrForCrosstalkSources_1031711_57.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/dccc054c-81f7-4450-aeac-cec6d907a509_isrForCrosstalkSources_1031711_57.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26eef18fd0f98e2735a73ecba4997fec with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e91f28f3-8aa2-4519-a222-52f7639bc09a_isrForCrosstalkSources_1031706_44.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e91f28f3-8aa2-4519-a222-52f7639bc09a_isrForCrosstalkSources_1031706_44.stderr parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccce4e9604e982527404c17d17febbf4 with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c3945e4e-f3a0-4100-bacd-ef6dfb0f218d_isrForCrosstalkSources_1031722_43.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c3945e4e-f3a0-4100-bacd-ef6dfb0f218d_isrForCrosstalkSources_1031722_43.stderr parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a780e4a36211985332f5f54ec19c1410 with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/032fda4f-5e9f-454f-a28e-8c3ab1ed60fd_isrForCrosstalkSources_1031713_18.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/032fda4f-5e9f-454f-a28e-8c3ab1ed60fd_isrForCrosstalkSources_1031713_18.stderr parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 806512699f284bd039bccabb5f40fb1a with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7248e68c-e933-403a-8de8-4e5195ebe6b2_isrForCrosstalkSources_1031732_41.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7248e68c-e933-403a-8de8-4e5195ebe6b2_isrForCrosstalkSources_1031732_41.stderr parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9c5c0910e5a09268ef511134b7e86dc with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7cad4730-bfcb-412f-8e20-ec0d376d0023_isrForCrosstalkSources_1031735_31.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7cad4730-bfcb-412f-8e20-ec0d376d0023_isrForCrosstalkSources_1031735_31.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b33514fde644c6459e49c68e2cd433ad with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/13fe53f9-b620-4dda-99ff-99e7559679f4_isrForCrosstalkSources_1031728_53.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/13fe53f9-b620-4dda-99ff-99e7559679f4_isrForCrosstalkSources_1031728_53.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a69952cc65eab21e0b399ae30f6001e with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2b5e6f66-e0e0-46c4-a165-b417a3c9547c_isrForCrosstalkSources_1031704_3.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2b5e6f66-e0e0-46c4-a165-b417a3c9547c_isrForCrosstalkSources_1031704_3.stderr parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d3fe3353f7784158c274f79330c02d9 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/b025b3ae-8ba9-4d6e-b32f-1bc7aeed1d03_isrForCrosstalkSources_1031730_44.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/b025b3ae-8ba9-4d6e-b32f-1bc7aeed1d03_isrForCrosstalkSources_1031730_44.stderr parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49485a04643fb5560ec85df63992698c with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0d8074f3-4ee3-4052-90ab-b6bff094e87e_isrForCrosstalkSources_1031717_51.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0d8074f3-4ee3-4052-90ab-b6bff094e87e_isrForCrosstalkSources_1031717_51.stderr parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aab672f3d55ffb18cccd39397901ea9b with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e4efd48c-997d-4a86-8f1a-20e3dba11650_isrForCrosstalkSources_1031717_30.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e4efd48c-997d-4a86-8f1a-20e3dba11650_isrForCrosstalkSources_1031717_30.stderr parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b2c8aebf6f59df30110eaf47c5ec975 with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/5619983a-d257-47d6-90b9-47ec204d0afa_isrForCrosstalkSources_1031738_6.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/5619983a-d257-47d6-90b9-47ec204d0afa_isrForCrosstalkSources_1031738_6.stderr parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5726aef0eed1e30b31ac421c2a2b452c with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/4aab500c-3e8f-4e2f-9ce6-650cd5b5573e_isrForCrosstalkSources_1031725_36.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/4aab500c-3e8f-4e2f-9ce6-650cd5b5573e_isrForCrosstalkSources_1031725_36.stderr parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 622015be225400e7d996fd8834301fe4 with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c954491b-e03f-48e9-a281-4ced61bfa3c4_isrForCrosstalkSources_1031729_39.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c954491b-e03f-48e9-a281-4ced61bfa3c4_isrForCrosstalkSources_1031729_39.stderr parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a394671f4b064dbedf899f688b84215e with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b90831c0-5887-4e95-8c8c-d2a0c414931c_isrForCrosstalkSources_1031728_60.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b90831c0-5887-4e95-8c8c-d2a0c414931c_isrForCrosstalkSources_1031728_60.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c04844a2a7a1f8a470e892261c88839d with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/2686c88b-7c23-410f-989e-292515fc22a2_isrForCrosstalkSources_1031718_3.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/2686c88b-7c23-410f-989e-292515fc22a2_isrForCrosstalkSources_1031718_3.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f84047476676200ddebc2c2976750ab with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/255c6649-ebd6-4249-9707-122b3961f676_isrForCrosstalkSources_1031732_57.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/255c6649-ebd6-4249-9707-122b3961f676_isrForCrosstalkSources_1031732_57.stderr parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc0383171cb27e30bd4bf3a8ef4efb14 with result from task 911 parsl.dataflow.dflow DEBUG: Task 911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6a0c0a16-e9ca-4486-9e0a-e73b8d13871a_isrForCrosstalkSources_1031711_34.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6a0c0a16-e9ca-4486-9e0a-e73b8d13871a_isrForCrosstalkSources_1031711_34.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48827473ca2096d54feac76fc1641ab4 with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/71b354bf-0968-4dca-b3a3-79ef80d19c4a_isrForCrosstalkSources_1031709_55.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/71b354bf-0968-4dca-b3a3-79ef80d19c4a_isrForCrosstalkSources_1031709_55.stderr parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31d536d1e3b6c0c91c68dd90f4c846de with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/30c38e7a-31f8-4daf-9093-b1b7fb3f5de8_isrForCrosstalkSources_1031735_54.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/30c38e7a-31f8-4daf-9093-b1b7fb3f5de8_isrForCrosstalkSources_1031735_54.stderr parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9afbf19bbca6285949b365ab47ce6f8 with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1337471b-a2ac-4e25-8870-87d2b75d9a41_isrForCrosstalkSources_1031735_44.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1337471b-a2ac-4e25-8870-87d2b75d9a41_isrForCrosstalkSources_1031735_44.stderr parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95b4fba1f4a1c2bca94c52d1264219f1 with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e0393250-be73-4894-827a-1f083db5e4df_isrForCrosstalkSources_1031710_36.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e0393250-be73-4894-827a-1f083db5e4df_isrForCrosstalkSources_1031710_36.stderr parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57f53f6d488f701ce72f09ffab7e9bf2 with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/afa10d15-6fbd-4c03-8c37-e430c955295d_isrForCrosstalkSources_1031713_54.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/afa10d15-6fbd-4c03-8c37-e430c955295d_isrForCrosstalkSources_1031713_54.stderr parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d61db018d87517b8563983fd38000545 with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f2ed30fa-ea4c-4e14-9962-bc40935db797_isrForCrosstalkSources_1031709_46.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f2ed30fa-ea4c-4e14-9962-bc40935db797_isrForCrosstalkSources_1031709_46.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c060735e3b256736b6cab5d6da38b484 with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e08b847e-20b3-4228-8303-a3716be336fe_isrForCrosstalkSources_1031703_20.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e08b847e-20b3-4228-8303-a3716be336fe_isrForCrosstalkSources_1031703_20.stderr parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bcdc5512400cc677fe6edee3efc60ac with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4b5f9846-d165-4e37-b1a6-88a6221fd6a8_isrForCrosstalkSources_1031738_36.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4b5f9846-d165-4e37-b1a6-88a6221fd6a8_isrForCrosstalkSources_1031738_36.stderr parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c549d8b3210819f90c9fc14168f1f0d1 with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/c80a7cc9-a783-4a01-9d4f-69488553d85a_isrForCrosstalkSources_1031701_8.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/c80a7cc9-a783-4a01-9d4f-69488553d85a_isrForCrosstalkSources_1031701_8.stderr parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c63ad975960ede986aa698a7a818911 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/18d2a18b-30cf-42b3-b4b8-e438087af2f1_isrForCrosstalkSources_1031728_56.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/18d2a18b-30cf-42b3-b4b8-e438087af2f1_isrForCrosstalkSources_1031728_56.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a79be44a3120690b7cbd1860680b2798 with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f5cb15a0-99e3-4ce6-be47-447cadc94128_isrForCrosstalkSources_1031723_26.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f5cb15a0-99e3-4ce6-be47-447cadc94128_isrForCrosstalkSources_1031723_26.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79e260b6b6ac572d015aaa569f372f4d with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/7ba70e4e-e56f-4373-b9df-e59f40db0cba_isrForCrosstalkSources_1031715_58.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/7ba70e4e-e56f-4373-b9df-e59f40db0cba_isrForCrosstalkSources_1031715_58.stderr parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cfe9b90772520b75f280fc99f283069 with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2f47466c-1e5d-48cf-9480-418cbb34a1ea_isrForCrosstalkSources_1031709_23.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2f47466c-1e5d-48cf-9480-418cbb34a1ea_isrForCrosstalkSources_1031709_23.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b4bb4423a75458742b988d395b390e6 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/aad5a357-1a3f-4aa9-85c1-0718e0463303_isrForCrosstalkSources_1031719_36.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/aad5a357-1a3f-4aa9-85c1-0718e0463303_isrForCrosstalkSources_1031719_36.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 728abcedd5124a3406f6de3e3141d08a with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/1112a384-6567-492c-ae5d-e7cef503df14_isrForCrosstalkSources_1031702_21.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/1112a384-6567-492c-ae5d-e7cef503df14_isrForCrosstalkSources_1031702_21.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76203d915b933ff949e18e07b7fffe10 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/988ec25f-21a4-4780-a85f-31b144e36c6b_isrForCrosstalkSources_1031733_14.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/988ec25f-21a4-4780-a85f-31b144e36c6b_isrForCrosstalkSources_1031733_14.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed213c03abf51fca63585954d5d97de0 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/49a2996f-be64-45e9-8f09-e41001d79939_isrForCrosstalkSources_1031722_33.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/49a2996f-be64-45e9-8f09-e41001d79939_isrForCrosstalkSources_1031722_33.stderr parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a8f839099940c24373db79a0cb473c3 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4bca09c3-292f-4b32-82a0-3226084b0eec_isrForCrosstalkSources_1031738_19.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4bca09c3-292f-4b32-82a0-3226084b0eec_isrForCrosstalkSources_1031738_19.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9389003a9013c8f81efd13675ad2b15d with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/34ab5aec-4b0b-40fc-a0c0-3c259594e0e5_isrForCrosstalkSources_1031719_58.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/34ab5aec-4b0b-40fc-a0c0-3c259594e0e5_isrForCrosstalkSources_1031719_58.stderr parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb2f10a279e69df85958ef310904eb53 with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c377a161-6f72-4309-b775-e6389c4e106c_isrForCrosstalkSources_1031703_53.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/c377a161-6f72-4309-b775-e6389c4e106c_isrForCrosstalkSources_1031703_53.stderr parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ce2e9756fd6c7d99dcc0191c9501aa6 with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ad117bcc-0838-433b-993e-563a9ef4116f_isrForCrosstalkSources_1031731_56.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ad117bcc-0838-433b-993e-563a9ef4116f_isrForCrosstalkSources_1031731_56.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 475b7b4f383d26f68ca1d2ec6d5a37b8 with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/59990dfb-12fc-4ebe-8835-d3d9d5cdf2dd_isrForCrosstalkSources_1031702_53.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/59990dfb-12fc-4ebe-8835-d3d9d5cdf2dd_isrForCrosstalkSources_1031702_53.stderr 2024-03-17 04:05:35 proc_lsst.multi:170 [INFO] [multi] job_id 17148064 proc_lsst.multi INFO: [multi] job_id 17148064 2024-03-17 04:05:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 63 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 63 2024-03-17 04:05:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148064 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148064 parsl.executors.status_handling DEBUG: Launched block 118 on executor multi with job ID 17148064 parsl.executors.status_handling INFO: Allocated block ID 119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-119 2024-03-17 04:05:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 2024-03-17 04:05:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 63 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 63 2024-03-17 04:05:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ee798017e3983b16365ccec4b70a112 with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8e75c1a5-85de-48fa-90b5-2b7c8224e0e3_isrForCrosstalkSources_1031735_50.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8e75c1a5-85de-48fa-90b5-2b7c8224e0e3_isrForCrosstalkSources_1031735_50.stderr parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5a764ea79251d305e4da4dd250dd6c1 with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/dde79f33-6b40-49b0-9804-305ca8939b2a_isrForCrosstalkSources_1031701_35.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/dde79f33-6b40-49b0-9804-305ca8939b2a_isrForCrosstalkSources_1031701_35.stderr parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51a03d5f6df854338ec95d1918115a05 with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c164b655-d69d-49f3-94e3-39a2faa55ba3_isrForCrosstalkSources_1031726_3.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c164b655-d69d-49f3-94e3-39a2faa55ba3_isrForCrosstalkSources_1031726_3.stderr parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13e87943a786a92916b47afe2c120405 with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2fd805e1-911e-4a26-8773-ebf2fea5fa35_isrForCrosstalkSources_1031731_62.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2fd805e1-911e-4a26-8773-ebf2fea5fa35_isrForCrosstalkSources_1031731_62.stderr parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5d1b5691e11f82d43b0fa860e254125 with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/c732f051-2857-4661-952b-059b5a5ef837_isrForCrosstalkSources_1031711_58.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/c732f051-2857-4661-952b-059b5a5ef837_isrForCrosstalkSources_1031711_58.stderr parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1891bbc9e05e026e189155d3ea2c7d52 with result from task 930 parsl.dataflow.dflow DEBUG: Task 930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6b2f5314-2be9-4661-920d-c49aae9f1892_isrForCrosstalkSources_1031716_14.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6b2f5314-2be9-4661-920d-c49aae9f1892_isrForCrosstalkSources_1031716_14.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0b18f557ff0f131822b2aed1f31a743 with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/991d3a8d-54c0-4553-96cc-8fd868a7c0a9_isrForCrosstalkSources_1031699_48.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/991d3a8d-54c0-4553-96cc-8fd868a7c0a9_isrForCrosstalkSources_1031699_48.stderr parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08c8d9d7987d79209b7894e6391058a6 with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/54443b03-9fb4-4db6-927a-bcc84c685d64_isrForCrosstalkSources_1031727_35.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/54443b03-9fb4-4db6-927a-bcc84c685d64_isrForCrosstalkSources_1031727_35.stderr 2024-03-17 04:05:36 proc_lsst.multi:170 [INFO] [multi] job_id 17148065 proc_lsst.multi INFO: [multi] job_id 17148065 2024-03-17 04:05:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 64 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 64 2024-03-17 04:05:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148065 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148065 parsl.executors.status_handling DEBUG: Launched block 119 on executor multi with job ID 17148065 parsl.executors.status_handling INFO: Allocated block ID 120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-120 2024-03-17 04:05:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 2024-03-17 04:05:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 64 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 64 2024-03-17 04:05:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49a25de4fe084110abf7d808574f0e35 with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bbd295d8-d986-401e-a418-2c8841cbaa3b_isrForCrosstalkSources_1031734_20.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bbd295d8-d986-401e-a418-2c8841cbaa3b_isrForCrosstalkSources_1031734_20.stderr parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd0e775fe320c13d9342327d731f9105 with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/2a0697a5-1421-41fd-b448-e51f4881bb6c_isrForCrosstalkSources_1031719_24.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/2a0697a5-1421-41fd-b448-e51f4881bb6c_isrForCrosstalkSources_1031719_24.stderr parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93916435bac7e8b7b0d2f5c097006b30 with result from task 910 parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/080cfe3e-b47f-4265-a4ca-f7f9fcbb1838_isrForCrosstalkSources_1031724_12.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/080cfe3e-b47f-4265-a4ca-f7f9fcbb1838_isrForCrosstalkSources_1031724_12.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ce6e974e257143aa86007a8777c6250 with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/53a2a7f4-79f5-4f17-8be6-1845f6a93221_isrForCrosstalkSources_1031709_49.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/53a2a7f4-79f5-4f17-8be6-1845f6a93221_isrForCrosstalkSources_1031709_49.stderr parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7aee8aeba161bbba6d8107c15968b3c4 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c4cce60d-98a5-4173-b8fe-eb14e2e1bc97_isrForCrosstalkSources_1031702_4.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c4cce60d-98a5-4173-b8fe-eb14e2e1bc97_isrForCrosstalkSources_1031702_4.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edee339094a2ce6649b3b7fcd79523c6 with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0eae4419-4608-43e8-bc5c-db6611a1490a_isrForCrosstalkSources_1031712_27.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0eae4419-4608-43e8-bc5c-db6611a1490a_isrForCrosstalkSources_1031712_27.stderr 2024-03-17 04:05:37 proc_lsst.multi:170 [INFO] [multi] job_id 17148066 proc_lsst.multi INFO: [multi] job_id 17148066 2024-03-17 04:05:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 65 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 65 2024-03-17 04:05:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148066 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148066 parsl.executors.status_handling DEBUG: Launched block 120 on executor multi with job ID 17148066 parsl.executors.status_handling INFO: Allocated block ID 121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-121 2024-03-17 04:05:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 2024-03-17 04:05:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 65 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 65 2024-03-17 04:05:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d6371d0c6d24a66325a5b04ddf2e047 with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/26035f50-f7ab-4da8-95cf-ca6352c87e4a_isrForCrosstalkSources_1031733_3.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/26035f50-f7ab-4da8-95cf-ca6352c87e4a_isrForCrosstalkSources_1031733_3.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d064219fd40a6259510b60b874cc93d with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8c82ad7e-48bd-4cdc-af90-6a76b16ce429_isrForCrosstalkSources_1031736_17.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8c82ad7e-48bd-4cdc-af90-6a76b16ce429_isrForCrosstalkSources_1031736_17.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5df97f4b061e175e50840480b464384e with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c7e44c07-57b1-407c-92c6-11f2238d2735_isrForCrosstalkSources_1031707_3.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c7e44c07-57b1-407c-92c6-11f2238d2735_isrForCrosstalkSources_1031707_3.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9e2afd23bf83c3f27db4cd6b12d372f with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d647867d-1c44-4715-bc6d-ed5d405e1242_isrForCrosstalkSources_1031727_43.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d647867d-1c44-4715-bc6d-ed5d405e1242_isrForCrosstalkSources_1031727_43.stderr parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b47e878f769e75637b511d3288117a9 with result from task 941 parsl.dataflow.dflow DEBUG: Task 941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/f341cd9f-9072-4cb8-af7a-3d949bc1a5d5_isrForCrosstalkSources_1031724_58.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/f341cd9f-9072-4cb8-af7a-3d949bc1a5d5_isrForCrosstalkSources_1031724_58.stderr parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d24abee8be7ab39d1c85a7a702888682 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/da6e23ae-dc7f-4c43-a23e-290af477c632_isrForCrosstalkSources_1031733_19.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/da6e23ae-dc7f-4c43-a23e-290af477c632_isrForCrosstalkSources_1031733_19.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f577addf3015526a38433f8a87e476db with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6a41316c-5c15-41f8-910b-7b9ade84d89c_isrForCrosstalkSources_1031726_17.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6a41316c-5c15-41f8-910b-7b9ade84d89c_isrForCrosstalkSources_1031726_17.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a03de97c1ab1d070a389fa3430ee92e with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c56669a4-d18e-4543-8158-a8eafd2b5dfb_isrForCrosstalkSources_1031710_9.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c56669a4-d18e-4543-8158-a8eafd2b5dfb_isrForCrosstalkSources_1031710_9.stderr parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e093a0b14390742428015e1a492df85f with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/333f367c-94cf-410f-abf0-bd2f2878c297_isrForCrosstalkSources_1031718_42.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/333f367c-94cf-410f-abf0-bd2f2878c297_isrForCrosstalkSources_1031718_42.stderr parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3bbb7974022adc343addbbd09014660 with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/49e67eef-5289-476b-b05b-c54628a7d016_isrForCrosstalkSources_1031724_37.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/49e67eef-5289-476b-b05b-c54628a7d016_isrForCrosstalkSources_1031724_37.stderr parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00204200000aea0d31a082e74d3994a3 with result from task 946 parsl.dataflow.dflow DEBUG: Task 946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e898f0e0-743a-4c76-8a72-3626fc4167ff_isrForCrosstalkSources_1031709_4.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e898f0e0-743a-4c76-8a72-3626fc4167ff_isrForCrosstalkSources_1031709_4.stderr parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 049e377eda42bda8d963de6d582f8fc3 with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/d9168d6a-3af6-40e8-af4b-3514ed7be3f2_isrForCrosstalkSources_1031710_43.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/d9168d6a-3af6-40e8-af4b-3514ed7be3f2_isrForCrosstalkSources_1031710_43.stderr parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 897b60bffe047f2009267e1218ff7716 with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/627c9a24-29ee-40bf-a852-733c1c57854f_isrForCrosstalkSources_1031715_49.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/627c9a24-29ee-40bf-a852-733c1c57854f_isrForCrosstalkSources_1031715_49.stderr 2024-03-17 04:05:38 proc_lsst.multi:170 [INFO] [multi] job_id 17148067 proc_lsst.multi INFO: [multi] job_id 17148067 2024-03-17 04:05:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 66 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 66 2024-03-17 04:05:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148067 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148067 parsl.executors.status_handling DEBUG: Launched block 121 on executor multi with job ID 17148067 parsl.executors.status_handling INFO: Allocated block ID 122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-122 2024-03-17 04:05:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 2024-03-17 04:05:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 66 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 66 2024-03-17 04:05:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 571ad006e1b22ce70ac5ba896765e209 with result from task 948 parsl.dataflow.dflow DEBUG: Task 948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/04360d5e-019e-4539-8afd-e5c3782c6223_isrForCrosstalkSources_1031715_53.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/04360d5e-019e-4539-8afd-e5c3782c6223_isrForCrosstalkSources_1031715_53.stderr parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d559df4e14931168327be078ff6360d6 with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/34e48b3d-1eea-4ef6-b8f1-6efcb43c93ee_isrForCrosstalkSources_1031735_61.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/34e48b3d-1eea-4ef6-b8f1-6efcb43c93ee_isrForCrosstalkSources_1031735_61.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65900c23717323f36f5fc6f9b3973935 with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/93968d73-4d2d-481d-a12e-8cb77c9f0a82_isrForCrosstalkSources_1031702_50.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/93968d73-4d2d-481d-a12e-8cb77c9f0a82_isrForCrosstalkSources_1031702_50.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcc9ec2d5b399db59fd090de87345a8e with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/49cafed2-3169-4dd6-83f3-1ed8561cf9c5_isrForCrosstalkSources_1031704_19.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/49cafed2-3169-4dd6-83f3-1ed8561cf9c5_isrForCrosstalkSources_1031704_19.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5b464d08ea2eb150e85c74b82775e45 with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7ad7794c-d3d2-4798-9487-710c9af00e4d_isrForCrosstalkSources_1031732_60.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7ad7794c-d3d2-4798-9487-710c9af00e4d_isrForCrosstalkSources_1031732_60.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8071582ff651bae6aff56a082dba5af8 with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/38bc3794-d8c0-4744-a59c-c72fd5a530ce_isrForCrosstalkSources_1031723_50.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/38bc3794-d8c0-4744-a59c-c72fd5a530ce_isrForCrosstalkSources_1031723_50.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 428fb5a8e203cf1d2b7e4e4a9f00308f with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/d39377b4-851c-4642-aea6-582a07c69914_isrForCrosstalkSources_1031723_61.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/d39377b4-851c-4642-aea6-582a07c69914_isrForCrosstalkSources_1031723_61.stderr 2024-03-17 04:05:39 proc_lsst.multi:170 [INFO] [multi] job_id 17148068 proc_lsst.multi INFO: [multi] job_id 17148068 2024-03-17 04:05:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 67 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 67 2024-03-17 04:05:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148068 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148068 parsl.executors.status_handling DEBUG: Launched block 122 on executor multi with job ID 17148068 parsl.executors.status_handling INFO: Allocated block ID 123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-123 2024-03-17 04:05:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 2024-03-17 04:05:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 67 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 67 2024-03-17 04:05:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 209076351ac61a107000cede699fed68 with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f9cc3402-09c8-4caa-a53b-f99f28b442ea_isrForCrosstalkSources_1031710_45.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f9cc3402-09c8-4caa-a53b-f99f28b442ea_isrForCrosstalkSources_1031710_45.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 256e49243eb732d9816953adbc8eda0a with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/26b6e16e-d899-4afb-9a8d-3d1ba4f0906c_isrForCrosstalkSources_1031727_5.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/26b6e16e-d899-4afb-9a8d-3d1ba4f0906c_isrForCrosstalkSources_1031727_5.stderr parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81edf8ea738f5514c2a0b991e58f53b5 with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/ec953354-1016-4430-a8c4-df63a00e46bd_isrForCrosstalkSources_1031707_56.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/ec953354-1016-4430-a8c4-df63a00e46bd_isrForCrosstalkSources_1031707_56.stderr parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd63054fb28c2ee4985152b980b9df4 with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d69b9c09-eedf-451a-bb63-27a591901f4f_isrForCrosstalkSources_1031733_1.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d69b9c09-eedf-451a-bb63-27a591901f4f_isrForCrosstalkSources_1031733_1.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dda3397b105e7e224febead7c5ea655f with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e2012346-8d63-467c-93bb-81b9f93d1440_isrForCrosstalkSources_1031706_47.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e2012346-8d63-467c-93bb-81b9f93d1440_isrForCrosstalkSources_1031706_47.stderr 2024-03-17 04:05:40 proc_lsst.multi:170 [INFO] [multi] job_id 17148069 proc_lsst.multi INFO: [multi] job_id 17148069 2024-03-17 04:05:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 68 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 68 2024-03-17 04:05:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148069 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148069 parsl.executors.status_handling DEBUG: Launched block 123 on executor multi with job ID 17148069 parsl.executors.status_handling INFO: Allocated block ID 124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-124 2024-03-17 04:05:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 2024-03-17 04:05:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 68 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 68 2024-03-17 04:05:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f58a7bee3944b30cc8580815012f16ae with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/50c87e5f-55ed-4422-8e29-d86ecbc50763_isrForCrosstalkSources_1031733_35.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/50c87e5f-55ed-4422-8e29-d86ecbc50763_isrForCrosstalkSources_1031733_35.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7415217b884ad24cd70e774d08671594 with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2e9cc4bb-dcdd-443b-9dfd-c4b950d726aa_isrForCrosstalkSources_1031704_5.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2e9cc4bb-dcdd-443b-9dfd-c4b950d726aa_isrForCrosstalkSources_1031704_5.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35ada41cef3a326ef313c2e939cc5446 with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c318b420-6301-4be4-a1c8-d35c3f104942_isrForCrosstalkSources_1031712_16.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c318b420-6301-4be4-a1c8-d35c3f104942_isrForCrosstalkSources_1031712_16.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c329a8266d3891ea25d0ac6e18fa98e with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c6ee71f7-6fc4-4422-a041-227dc2cfa73e_isrForCrosstalkSources_1031706_52.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c6ee71f7-6fc4-4422-a041-227dc2cfa73e_isrForCrosstalkSources_1031706_52.stderr parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b909b62f3ecf1674f7868a70f4b11757 with result from task 945 parsl.dataflow.dflow DEBUG: Task 945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/6400c591-3774-4497-a0c3-eda795fec70e_isrForCrosstalkSources_1031706_32.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/6400c591-3774-4497-a0c3-eda795fec70e_isrForCrosstalkSources_1031706_32.stderr 2024-03-17 04:05:41 proc_lsst.multi:170 [INFO] [multi] job_id 17148070 proc_lsst.multi INFO: [multi] job_id 17148070 2024-03-17 04:05:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 69 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 69 2024-03-17 04:05:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148070 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148070 parsl.executors.status_handling DEBUG: Launched block 124 on executor multi with job ID 17148070 parsl.executors.status_handling INFO: Allocated block ID 125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-125 2024-03-17 04:05:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 2024-03-17 04:05:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 69 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 69 2024-03-17 04:05:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38c2e5b0a6f54718551aa454f67097f3 with result from task 942 parsl.dataflow.dflow DEBUG: Task 942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/222d355a-e1ab-4be2-b9db-f5d218caccbf_isrForCrosstalkSources_1031725_40.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/222d355a-e1ab-4be2-b9db-f5d218caccbf_isrForCrosstalkSources_1031725_40.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63469c8b9271018caccf1baa87e14df7 with result from task 938 parsl.dataflow.dflow DEBUG: Task 938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/85029dc4-d5b8-4117-a51f-b8b2bb2562ea_isrForCrosstalkSources_1031707_40.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/85029dc4-d5b8-4117-a51f-b8b2bb2562ea_isrForCrosstalkSources_1031707_40.stderr parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry babe61840006c3e2983eb0e0502b1c80 with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d8cca335-01a4-4d31-8421-867125c58163_isrForCrosstalkSources_1031728_21.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d8cca335-01a4-4d31-8421-867125c58163_isrForCrosstalkSources_1031728_21.stderr parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9061a9f28a858d8e6790103b0b604b7e with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/1cc48962-9f3b-4cf9-b75a-ace6efb56350_isrForCrosstalkSources_1031722_14.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/1cc48962-9f3b-4cf9-b75a-ace6efb56350_isrForCrosstalkSources_1031722_14.stderr parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccc27f7fba65d902eb31e1a23d334698 with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/82a9fc08-279b-4ea2-ac0a-ddeaf37bd650_isrForCrosstalkSources_1031732_46.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/82a9fc08-279b-4ea2-ac0a-ddeaf37bd650_isrForCrosstalkSources_1031732_46.stderr parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c48f2340bcaf23204fadad5a264284a with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d51c18e4-ccd3-42ae-b5a8-5e1ab60492e5_isrForCrosstalkSources_1031737_55.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d51c18e4-ccd3-42ae-b5a8-5e1ab60492e5_isrForCrosstalkSources_1031737_55.stderr parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7ca833d3cc732cd11468dcbf4f0434f with result from task 947 parsl.dataflow.dflow DEBUG: Task 947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/9bb5b6c8-d08b-442a-b44b-b38c5a34ca72_isrForCrosstalkSources_1031735_51.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/9bb5b6c8-d08b-442a-b44b-b38c5a34ca72_isrForCrosstalkSources_1031735_51.stderr parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94259cced131fbb64cb4519337faf92e with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/eab76665-f406-4592-9cd3-ee89d2d41ea7_isrForCrosstalkSources_1031701_38.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/eab76665-f406-4592-9cd3-ee89d2d41ea7_isrForCrosstalkSources_1031701_38.stderr parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90214b810d089dd7e65188e36b80a394 with result from task 978 parsl.dataflow.dflow DEBUG: Task 978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7d32467c-e089-4318-8aeb-8a02ba447397_isrForCrosstalkSources_1031702_14.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7d32467c-e089-4318-8aeb-8a02ba447397_isrForCrosstalkSources_1031702_14.stderr parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89ecdb485fd5926889b366dc4add9833 with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c72ebb1e-0a34-412a-aa10-d63cd3321ec2_isrForCrosstalkSources_1031728_54.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c72ebb1e-0a34-412a-aa10-d63cd3321ec2_isrForCrosstalkSources_1031728_54.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f3fb1ea59c38a16e0a95658d7035eff with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a8d41e6e-c2bf-46ea-9478-e9bd95eb70cf_isrForCrosstalkSources_1031721_57.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a8d41e6e-c2bf-46ea-9478-e9bd95eb70cf_isrForCrosstalkSources_1031721_57.stderr parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c110d4f4c0eda7144f12d90c0d30158f with result from task 975 parsl.dataflow.dflow DEBUG: Task 975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d48cfc72-f424-476a-8fe8-503c9def50e4_isrForCrosstalkSources_1031738_59.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d48cfc72-f424-476a-8fe8-503c9def50e4_isrForCrosstalkSources_1031738_59.stderr parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4e0cf1b8b4cef2f7e7403b919362307 with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cc96b6fb-f930-4e6b-8c70-8f79c91354fd_isrForCrosstalkSources_1031716_52.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cc96b6fb-f930-4e6b-8c70-8f79c91354fd_isrForCrosstalkSources_1031716_52.stderr parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f46e6a80f47b4714ecb0ab4c00cf0550 with result from task 944 parsl.dataflow.dflow DEBUG: Task 944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a5a9b035-5e52-4643-bff3-f081c90e799a_isrForCrosstalkSources_1031728_6.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a5a9b035-5e52-4643-bff3-f081c90e799a_isrForCrosstalkSources_1031728_6.stderr parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fb61ebe3add6f2dc7f9a377cd974daa with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3964ac24-3d2c-42ff-a417-e705f39c3324_isrForCrosstalkSources_1031738_51.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3964ac24-3d2c-42ff-a417-e705f39c3324_isrForCrosstalkSources_1031738_51.stderr parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0a3c563418b5ff391fa4c4d7e318ee4 with result from task 966 parsl.dataflow.dflow DEBUG: Task 966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/84171944-cec1-4b9e-bbf4-35b8654f0f38_isrForCrosstalkSources_1031705_34.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/84171944-cec1-4b9e-bbf4-35b8654f0f38_isrForCrosstalkSources_1031705_34.stderr parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04d64e7d1bae057de45e73651d13123b with result from task 949 parsl.dataflow.dflow DEBUG: Task 949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/26906c93-5297-4d21-b505-8dcf71f6a2df_isrForCrosstalkSources_1031713_45.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/26906c93-5297-4d21-b505-8dcf71f6a2df_isrForCrosstalkSources_1031713_45.stderr 2024-03-17 04:05:42 proc_lsst.multi:170 [INFO] [multi] job_id 17148071 proc_lsst.multi INFO: [multi] job_id 17148071 2024-03-17 04:05:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 70 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 70 2024-03-17 04:05:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148071 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148071 parsl.executors.status_handling DEBUG: Launched block 125 on executor multi with job ID 17148071 parsl.executors.status_handling INFO: Allocated block ID 126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-126 2024-03-17 04:05:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 2024-03-17 04:05:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 70 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 70 2024-03-17 04:05:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9482d93d3f840c1306524ef7c5f2affc with result from task 1015 parsl.dataflow.dflow DEBUG: Task 1015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/31b593b0-85be-4bed-bac5-1fb41bdd2d7e_isrForCrosstalkSources_1031713_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/31b593b0-85be-4bed-bac5-1fb41bdd2d7e_isrForCrosstalkSources_1031713_14.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccd97cfe36d2ce6ee3a7b9e6c2e2f130 with result from task 982 parsl.dataflow.dflow DEBUG: Task 982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/9846362d-eed9-475b-96b3-2a40d0b87187_isrForCrosstalkSources_1031717_45.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/9846362d-eed9-475b-96b3-2a40d0b87187_isrForCrosstalkSources_1031717_45.stderr parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 036b3d390d90b3464c2b1728b0f19783 with result from task 952 parsl.dataflow.dflow DEBUG: Task 952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4c5c9a89-931e-4b67-a3a0-17cff2e290ae_isrForCrosstalkSources_1031732_35.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/4c5c9a89-931e-4b67-a3a0-17cff2e290ae_isrForCrosstalkSources_1031732_35.stderr parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry deefca112cbae3909002a341cf25ec0f with result from task 1025 parsl.dataflow.dflow DEBUG: Task 1025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/67c2826a-0186-4f0a-bced-15a0ecff34ab_isrForCrosstalkSources_1031715_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/67c2826a-0186-4f0a-bced-15a0ecff34ab_isrForCrosstalkSources_1031715_4.stderr parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce852a61204134eb943061bced57a61d with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a2ef7ec3-ff79-48e1-ba7d-c11d21e82401_isrForCrosstalkSources_1031736_9.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a2ef7ec3-ff79-48e1-ba7d-c11d21e82401_isrForCrosstalkSources_1031736_9.stderr parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d100b113725cb743cba825fce98e05bb with result from task 1027 parsl.dataflow.dflow DEBUG: Task 1027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/1bec8cd7-9eda-43c6-9480-79ed701f8ee8_isrForCrosstalkSources_1031734_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/1bec8cd7-9eda-43c6-9480-79ed701f8ee8_isrForCrosstalkSources_1031734_3.stderr parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9058c291fee2a18e9a9bd65060bf8c1f with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e17821aa-8a47-4938-850b-b3b23a960142_isrForCrosstalkSources_1031709_5.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/e17821aa-8a47-4938-850b-b3b23a960142_isrForCrosstalkSources_1031709_5.stderr parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b6218cbb4720dfb539d547feeefa021 with result from task 1045 parsl.dataflow.dflow DEBUG: Task 1045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/acbe27f6-cda5-43e2-be00-ed8d1529ae22_isrForCrosstalkSources_1031699_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/acbe27f6-cda5-43e2-be00-ed8d1529ae22_isrForCrosstalkSources_1031699_9.stderr parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c53b0086550bb3c7922568238d14dbef with result from task 1039 parsl.dataflow.dflow DEBUG: Task 1039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/bfbf3363-c933-4b29-aec8-7991048053d4_isrForCrosstalkSources_1031727_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/bfbf3363-c933-4b29-aec8-7991048053d4_isrForCrosstalkSources_1031727_9.stderr parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea9c187629dead6baff764c7d59523cd with result from task 959 parsl.dataflow.dflow DEBUG: Task 959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e5791306-2b01-45a6-a20b-729a3b81de9f_isrForCrosstalkSources_1031710_46.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e5791306-2b01-45a6-a20b-729a3b81de9f_isrForCrosstalkSources_1031710_46.stderr parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78a8ebec154da3e3c5f5aaebea486df0 with result from task 1028 parsl.dataflow.dflow DEBUG: Task 1028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/81835bc2-ea10-4bca-9a14-fb4d051fceb1_isrForCrosstalkSources_1031706_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/81835bc2-ea10-4bca-9a14-fb4d051fceb1_isrForCrosstalkSources_1031706_16.stderr parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 839cc6871b2eecadb78243d6476636d8 with result from task 956 parsl.dataflow.dflow DEBUG: Task 956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/dceac69a-183d-4a35-80ab-5119b4a87a27_isrForCrosstalkSources_1031730_8.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/dceac69a-183d-4a35-80ab-5119b4a87a27_isrForCrosstalkSources_1031730_8.stderr parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51d63265211c5ef91788b0c26f939c38 with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7cb3d3f7-609b-45f5-89e2-202d19be5b42_isrForCrosstalkSources_1031704_36.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7cb3d3f7-609b-45f5-89e2-202d19be5b42_isrForCrosstalkSources_1031704_36.stderr parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4553f923192edae12676ed9891bff432 with result from task 951 parsl.dataflow.dflow DEBUG: Task 951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/091f5583-93e3-4a9b-b98b-7b488d7e371c_isrForCrosstalkSources_1031719_42.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/091f5583-93e3-4a9b-b98b-7b488d7e371c_isrForCrosstalkSources_1031719_42.stderr parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62c8bbe58a5c6e5ffc9340088c5bf0b7 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/19ab9b10-f6a3-403f-96c8-49094a23272f_isrForCrosstalkSources_1031705_19.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/19ab9b10-f6a3-403f-96c8-49094a23272f_isrForCrosstalkSources_1031705_19.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81f21160380c1bf57e90bdcaa8558d43 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/829eed61-9181-4f19-9ece-e3c38a80b04e_isrForCrosstalkSources_1031736_13.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/829eed61-9181-4f19-9ece-e3c38a80b04e_isrForCrosstalkSources_1031736_13.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf3d91928345e50dea688544370e7ba8 with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/36ba444f-4f6f-46f9-8f08-face41e73143_isrForCrosstalkSources_1031704_30.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/36ba444f-4f6f-46f9-8f08-face41e73143_isrForCrosstalkSources_1031704_30.stderr 2024-03-17 04:05:43 proc_lsst.multi:170 [INFO] [multi] job_id 17148072 proc_lsst.multi INFO: [multi] job_id 17148072 2024-03-17 04:05:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 71 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 71 2024-03-17 04:05:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148072 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148072 parsl.executors.status_handling DEBUG: Launched block 126 on executor multi with job ID 17148072 parsl.executors.status_handling INFO: Allocated block ID 127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-127 2024-03-17 04:05:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 2024-03-17 04:05:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 71 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 71 2024-03-17 04:05:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2911c03442b5d689c47d325c23dd4f7f with result from task 1036 parsl.dataflow.dflow DEBUG: Task 1036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/4c1e4e4b-6977-4306-b210-90f57c78b4f5_isrForCrosstalkSources_1031700_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/4c1e4e4b-6977-4306-b210-90f57c78b4f5_isrForCrosstalkSources_1031700_20.stderr parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 558e0a65dc0adf616e318ddfba2801b7 with result from task 1037 parsl.dataflow.dflow DEBUG: Task 1037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/abe372b6-7f14-4322-a586-72e4793cb40d_isrForCrosstalkSources_1031708_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/abe372b6-7f14-4322-a586-72e4793cb40d_isrForCrosstalkSources_1031708_29.stderr parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91be56eed9a7b77b813b53fc9824edb8 with result from task 1041 parsl.dataflow.dflow DEBUG: Task 1041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/5bf2a9b2-eec9-4a06-b724-cb4c42c49d10_isrForCrosstalkSources_1031710_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/5bf2a9b2-eec9-4a06-b724-cb4c42c49d10_isrForCrosstalkSources_1031710_55.stderr parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab84a578e25936c51ceaefd8ad6bd5a5 with result from task 933 parsl.dataflow.dflow DEBUG: Task 933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/f30b31a7-9b3a-4d90-bca2-ac9ffd273efd_isrForCrosstalkSources_1031708_36.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/f30b31a7-9b3a-4d90-bca2-ac9ffd273efd_isrForCrosstalkSources_1031708_36.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22b03b6f57cb256f42b358ab9cc7108c with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/89b00491-c389-402c-a3e8-5f025b192999_isrForCrosstalkSources_1031712_37.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/89b00491-c389-402c-a3e8-5f025b192999_isrForCrosstalkSources_1031712_37.stderr parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 320808c182217785b23536e5b05d487b with result from task 967 parsl.dataflow.dflow DEBUG: Task 967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/97497fc1-37da-4fa3-8a5f-bcdc2ef5cf0e_isrForCrosstalkSources_1031717_9.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/97497fc1-37da-4fa3-8a5f-bcdc2ef5cf0e_isrForCrosstalkSources_1031717_9.stderr parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22beb60820b4861e115218d981ed4629 with result from task 981 parsl.dataflow.dflow DEBUG: Task 981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ee00135e-4446-439e-9f55-ee493a0bef58_isrForCrosstalkSources_1031737_8.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ee00135e-4446-439e-9f55-ee493a0bef58_isrForCrosstalkSources_1031737_8.stderr parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06a2e82afbbd0f4678837f4ba88c2563 with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4a3029ae-1d01-4103-bfb4-c44e86eec0fd_isrForCrosstalkSources_1031718_35.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4a3029ae-1d01-4103-bfb4-c44e86eec0fd_isrForCrosstalkSources_1031718_35.stderr parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 140c21f80884bdf7dd19b9fa5423d4a7 with result from task 943 parsl.dataflow.dflow DEBUG: Task 943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/fa69d29d-9890-41a0-a319-529db4917e9a_isrForCrosstalkSources_1031721_12.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/fa69d29d-9890-41a0-a319-529db4917e9a_isrForCrosstalkSources_1031721_12.stderr parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b6b3ab7775ec7c713c5a6b36db15956 with result from task 979 parsl.dataflow.dflow DEBUG: Task 979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/392913c0-ae88-42b2-9e12-727daf5976e3_isrForCrosstalkSources_1031700_14.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/392913c0-ae88-42b2-9e12-727daf5976e3_isrForCrosstalkSources_1031700_14.stderr parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee0cff02c8757dbb0edb079c3d1d3133 with result from task 973 parsl.dataflow.dflow DEBUG: Task 973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/544a33b2-f1a8-462b-abb3-dd385ebc1af9_isrForCrosstalkSources_1031721_19.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/544a33b2-f1a8-462b-abb3-dd385ebc1af9_isrForCrosstalkSources_1031721_19.stderr parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ea8a3572b228b290fe342587a0d2a0a with result from task 934 parsl.dataflow.dflow DEBUG: Task 934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/63fd52fd-25cc-4156-bd17-c6517aa3c0f3_isrForCrosstalkSources_1031712_38.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/63fd52fd-25cc-4156-bd17-c6517aa3c0f3_isrForCrosstalkSources_1031712_38.stderr parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fccad5f96652bd7d78f9b5b630cc660 with result from task 974 parsl.dataflow.dflow DEBUG: Task 974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1d445ddc-c885-4a59-958a-a446a82a599a_isrForCrosstalkSources_1031733_53.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1d445ddc-c885-4a59-958a-a446a82a599a_isrForCrosstalkSources_1031733_53.stderr parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f2fa0563fc2bb07ade144243dda1b1e with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/649e1403-ffd1-41af-9c44-5899eff97e0c_isrForCrosstalkSources_1031733_41.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/649e1403-ffd1-41af-9c44-5899eff97e0c_isrForCrosstalkSources_1031733_41.stderr parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34ee2c97aee86f8088dbbc929980f96a with result from task 1030 parsl.dataflow.dflow DEBUG: Task 1030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/1e1482d4-d51f-40a2-aa51-8b8823ea4815_isrForCrosstalkSources_1031728_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/1e1482d4-d51f-40a2-aa51-8b8823ea4815_isrForCrosstalkSources_1031728_39.stderr 2024-03-17 04:05:45 proc_lsst.multi:170 [INFO] [multi] job_id 17148073 proc_lsst.multi INFO: [multi] job_id 17148073 2024-03-17 04:05:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 72 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 72 2024-03-17 04:05:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148073 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148073 parsl.executors.status_handling DEBUG: Launched block 127 on executor multi with job ID 17148073 parsl.executors.status_handling INFO: Allocated block ID 128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-128 2024-03-17 04:05:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 2024-03-17 04:05:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 72 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 72 2024-03-17 04:05:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81563c38bc518487bd2f3c4d123ae236 with result from task 957 parsl.dataflow.dflow DEBUG: Task 957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/680488e0-c568-426e-9398-efa51a89f87d_isrForCrosstalkSources_1031725_44.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/680488e0-c568-426e-9398-efa51a89f87d_isrForCrosstalkSources_1031725_44.stderr parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f9d153df79333922337a2a046c72146 with result from task 971 parsl.dataflow.dflow DEBUG: Task 971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/6c046e9d-d89a-4b88-b156-c0e1bc597007_isrForCrosstalkSources_1031731_32.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/6c046e9d-d89a-4b88-b156-c0e1bc597007_isrForCrosstalkSources_1031731_32.stderr parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24295cc9de4eef21c512f759597409ac with result from task 954 parsl.dataflow.dflow DEBUG: Task 954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c76310df-2ffc-403c-9f05-69373879a9f3_isrForCrosstalkSources_1031725_29.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/c76310df-2ffc-403c-9f05-69373879a9f3_isrForCrosstalkSources_1031725_29.stderr parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f620fbf1bbf3b37839dd57da7bbf0e6d with result from task 963 parsl.dataflow.dflow DEBUG: Task 963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/6ec9ebf4-add5-46e6-a37a-e4fc1bdff4d8_isrForCrosstalkSources_1031736_39.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/6ec9ebf4-add5-46e6-a37a-e4fc1bdff4d8_isrForCrosstalkSources_1031736_39.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5990d5a5264b42c824dbc1439e2cf8f with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0b6d726b-1726-4d48-97b2-7cccfc18bee1_isrForCrosstalkSources_1031714_41.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0b6d726b-1726-4d48-97b2-7cccfc18bee1_isrForCrosstalkSources_1031714_41.stderr parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 843c21253db619eda2781f4e42666e18 with result from task 953 parsl.dataflow.dflow DEBUG: Task 953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/8649c994-df28-40e4-9d50-c8365816e4c2_isrForCrosstalkSources_1031727_45.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/8649c994-df28-40e4-9d50-c8365816e4c2_isrForCrosstalkSources_1031727_45.stderr parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 397f4cac8edf6d9f4d8f0d9f3f7f8ca2 with result from task 940 parsl.dataflow.dflow DEBUG: Task 940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d0bdd09e-5a18-4264-9227-dffc9d379879_isrForCrosstalkSources_1031730_23.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d0bdd09e-5a18-4264-9227-dffc9d379879_isrForCrosstalkSources_1031730_23.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1246965ee9f65692fcccab0581c4e75 with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/226d99d8-7871-4f6c-8b6a-72e0e1ae0e20_isrForCrosstalkSources_1031706_27.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/226d99d8-7871-4f6c-8b6a-72e0e1ae0e20_isrForCrosstalkSources_1031706_27.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d75d09fe8d3f5c04e5343e5db09a2f07 with result from task 937 parsl.dataflow.dflow DEBUG: Task 937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8f389777-933f-49da-8952-e17bd2c65cce_isrForCrosstalkSources_1031732_61.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/8f389777-933f-49da-8952-e17bd2c65cce_isrForCrosstalkSources_1031732_61.stderr parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56e9ba392cf4d3fd63350d559bc90c1c with result from task 965 parsl.dataflow.dflow DEBUG: Task 965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/cc77e4f3-fe1c-4d8a-9b2e-bf33ab88ffbb_isrForCrosstalkSources_1031711_18.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/cc77e4f3-fe1c-4d8a-9b2e-bf33ab88ffbb_isrForCrosstalkSources_1031711_18.stderr parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef66498112ac5d1dc6a05bcdbc9a3ca6 with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/32594bca-6008-4b94-af84-607929ceacdd_isrForCrosstalkSources_1031709_13.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/32594bca-6008-4b94-af84-607929ceacdd_isrForCrosstalkSources_1031709_13.stderr parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12d7d9511755e160efca6f2de9fe8410 with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3d7bcc40-86c5-40fe-9798-df9af75d01c7_isrForCrosstalkSources_1031732_39.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/3d7bcc40-86c5-40fe-9798-df9af75d01c7_isrForCrosstalkSources_1031732_39.stderr parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a55f02889bff4a443225d9d929079579 with result from task 968 parsl.dataflow.dflow DEBUG: Task 968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/283a6368-6d93-4cb5-bde1-85506dd006a0_isrForCrosstalkSources_1031736_43.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/283a6368-6d93-4cb5-bde1-85506dd006a0_isrForCrosstalkSources_1031736_43.stderr parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a49dbde10f9a415fca09967d4ed9331 with result from task 1069 parsl.dataflow.dflow DEBUG: Task 1069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/dea7c380-4dd8-489a-aec6-e2a9889fe897_isrForCrosstalkSources_1031707_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/dea7c380-4dd8-489a-aec6-e2a9889fe897_isrForCrosstalkSources_1031707_25.stderr parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5adea3fc3e95f1fa00eb366e40ccbc98 with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/fc6c6241-936f-4723-9981-1c90ae7fafe3_isrForCrosstalkSources_1031725_50.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/fc6c6241-936f-4723-9981-1c90ae7fafe3_isrForCrosstalkSources_1031725_50.stderr parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43129cc3f8f6676f45b5853207284a20 with result from task 1051 parsl.dataflow.dflow DEBUG: Task 1051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/65cfbfae-0e0d-4c8b-9b3e-a12ede5ea8fc_isrForCrosstalkSources_1031737_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/65cfbfae-0e0d-4c8b-9b3e-a12ede5ea8fc_isrForCrosstalkSources_1031737_6.stderr parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55cb3a8a245bb1be98ad9b8ec8c39e46 with result from task 935 parsl.dataflow.dflow DEBUG: Task 935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2e83e6a2-2ee9-4dd7-a427-7ed315957cfb_isrForCrosstalkSources_1031736_30.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2e83e6a2-2ee9-4dd7-a427-7ed315957cfb_isrForCrosstalkSources_1031736_30.stderr parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d91d59ca2ea130a1737c9d53a009e668 with result from task 1033 parsl.dataflow.dflow DEBUG: Task 1033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fde7526e-92f0-492a-8faa-3dcdab7a63a8_isrForCrosstalkSources_1031729_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fde7526e-92f0-492a-8faa-3dcdab7a63a8_isrForCrosstalkSources_1031729_17.stderr parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19f258a4698482ca2451089f7e13aec9 with result from task 1066 parsl.dataflow.dflow DEBUG: Task 1066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/32eb516c-4702-40f6-af98-ea318518a5d1_isrForCrosstalkSources_1031701_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/32eb516c-4702-40f6-af98-ea318518a5d1_isrForCrosstalkSources_1031701_39.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e12c2d8c37c9b59329f134d2fa922859 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/951732c3-d88e-4de6-88d0-36818568b9d3_isrForCrosstalkSources_1031715_38.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/951732c3-d88e-4de6-88d0-36818568b9d3_isrForCrosstalkSources_1031715_38.stderr parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a586ba5f383d417f7a581a919a227b38 with result from task 985 parsl.dataflow.dflow DEBUG: Task 985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/b59921bd-e26a-49cd-affa-1f0e3cd99df7_isrForCrosstalkSources_1031737_11.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/b59921bd-e26a-49cd-affa-1f0e3cd99df7_isrForCrosstalkSources_1031737_11.stderr parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c79c2f0bd71d233f9febe89d0d38b175 with result from task 958 parsl.dataflow.dflow DEBUG: Task 958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9aa4793e-e063-4e19-b325-2ae0e41b5b19_isrForCrosstalkSources_1031733_46.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9aa4793e-e063-4e19-b325-2ae0e41b5b19_isrForCrosstalkSources_1031733_46.stderr 2024-03-17 04:05:46 proc_lsst.multi:170 [INFO] [multi] job_id 17148074 proc_lsst.multi INFO: [multi] job_id 17148074 2024-03-17 04:05:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 73 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 73 2024-03-17 04:05:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148074 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148074 parsl.executors.status_handling DEBUG: Launched block 128 on executor multi with job ID 17148074 parsl.executors.status_handling INFO: Allocated block ID 129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-129 2024-03-17 04:05:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 2024-03-17 04:05:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 73 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 73 2024-03-17 04:05:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39848a992cb9d36ac1c978ebb96213c5 with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/e2eea651-8d2b-49d5-85fe-dc4a6a963a42_isrForCrosstalkSources_1031737_54.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/e2eea651-8d2b-49d5-85fe-dc4a6a963a42_isrForCrosstalkSources_1031737_54.stderr parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1f352cdaf6f1ead9f254be7fbfdbbd5 with result from task 1000 parsl.dataflow.dflow DEBUG: Task 1000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1d2894bb-f2fa-43d6-9f49-8222a7b50e03_isrForCrosstalkSources_1031703_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1d2894bb-f2fa-43d6-9f49-8222a7b50e03_isrForCrosstalkSources_1031703_16.stderr parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9583742e92c17eda4955ff1842c362 with result from task 972 parsl.dataflow.dflow DEBUG: Task 972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/64c1863e-046e-4103-955e-9ab0ae3aa435_isrForCrosstalkSources_1031707_35.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/64c1863e-046e-4103-955e-9ab0ae3aa435_isrForCrosstalkSources_1031707_35.stderr parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2fbf46e70993a059145f7fecbb22ec7 with result from task 1034 parsl.dataflow.dflow DEBUG: Task 1034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c11a230f-a8dd-4253-a521-47dc6dc2da6f_isrForCrosstalkSources_1031732_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c11a230f-a8dd-4253-a521-47dc6dc2da6f_isrForCrosstalkSources_1031732_53.stderr parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c559b0006c8f935a1e26df2941ccbf2 with result from task 976 parsl.dataflow.dflow DEBUG: Task 976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f2c9145d-f822-44dd-9310-20eafccda547_isrForCrosstalkSources_1031721_45.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f2c9145d-f822-44dd-9310-20eafccda547_isrForCrosstalkSources_1031721_45.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4ced211059ee7bc93ad986a60873d9c with result from task 936 parsl.dataflow.dflow DEBUG: Task 936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/beea3b8e-6cee-4ee1-b9c9-5125125bd487_isrForCrosstalkSources_1031701_53.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/beea3b8e-6cee-4ee1-b9c9-5125125bd487_isrForCrosstalkSources_1031701_53.stderr parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad9407f7d2d50af6ff179892a2d49b71 with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/119ca187-d759-41f0-8606-ebba9416fa0a_isrForCrosstalkSources_1031724_45.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/119ca187-d759-41f0-8606-ebba9416fa0a_isrForCrosstalkSources_1031724_45.stderr parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13c1f0ad5e0c433b54a1e66cc8534eb9 with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/85fff9bb-eaa2-489a-9785-f46f8f6224ec_isrForCrosstalkSources_1031735_14.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/85fff9bb-eaa2-489a-9785-f46f8f6224ec_isrForCrosstalkSources_1031735_14.stderr parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34e1037ce1d56e11c073123fd7d45c2c with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/49fc1b62-fada-4cdc-8f97-cfa66cb182f1_isrForCrosstalkSources_1031738_58.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/49fc1b62-fada-4cdc-8f97-cfa66cb182f1_isrForCrosstalkSources_1031738_58.stderr parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 329509e7bfdf45c3309f91a4cc1e56f7 with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4cfda397-da65-40e9-94b3-7f21fa19e298_isrForCrosstalkSources_1031709_29.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4cfda397-da65-40e9-94b3-7f21fa19e298_isrForCrosstalkSources_1031709_29.stderr parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8a2d7d1ae75d1e00729650218ae353a with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f38b6f34-b696-46ed-8977-de68c84cd252_isrForCrosstalkSources_1031699_61.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f38b6f34-b696-46ed-8977-de68c84cd252_isrForCrosstalkSources_1031699_61.stderr parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f6bb61d95f0baf6bb080a0fd85d3dc2 with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/263cc0a7-a2eb-498f-b260-6b531aba9d70_isrForCrosstalkSources_1031729_40.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/263cc0a7-a2eb-498f-b260-6b531aba9d70_isrForCrosstalkSources_1031729_40.stderr parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2b38fa59c39278517c0f08ef0558a58 with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d73e8676-6af5-4ee7-8170-2927c3a3526a_isrForCrosstalkSources_1031728_1.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/d73e8676-6af5-4ee7-8170-2927c3a3526a_isrForCrosstalkSources_1031728_1.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ac0406ac135652bd96dd3cbd8caf019 with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/6276af7a-37fc-4447-bbd5-80b558c26789_isrForCrosstalkSources_1031702_27.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/6276af7a-37fc-4447-bbd5-80b558c26789_isrForCrosstalkSources_1031702_27.stderr parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d70c940c36c97170c11980eae9954052 with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/c8519919-774a-45b5-b93f-122021ce7c04_isrForCrosstalkSources_1031733_23.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/c8519919-774a-45b5-b93f-122021ce7c04_isrForCrosstalkSources_1031733_23.stderr parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 184c2974c077e07de341fc3b1e7ea40a with result from task 980 parsl.dataflow.dflow DEBUG: Task 980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/40cd924e-ff8d-4c1b-a468-669d7618eb9a_isrForCrosstalkSources_1031710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/40cd924e-ff8d-4c1b-a468-669d7618eb9a_isrForCrosstalkSources_1031710_61.stderr parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c12a36ac3cd3bd80b3079db251af06b with result from task 1057 parsl.dataflow.dflow DEBUG: Task 1057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bdf87c6b-1743-477e-a5a2-0dd693e2f15b_isrForCrosstalkSources_1031734_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bdf87c6b-1743-477e-a5a2-0dd693e2f15b_isrForCrosstalkSources_1031734_18.stderr parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 021d8922db0ae05bf92025be5ea9b5f8 with result from task 962 parsl.dataflow.dflow DEBUG: Task 962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/439fce95-a1e9-46b8-96b2-6d87776d364f_isrForCrosstalkSources_1031716_57.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/439fce95-a1e9-46b8-96b2-6d87776d364f_isrForCrosstalkSources_1031716_57.stderr parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d541584212f5e181a8e306ff704cdbc with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/a309025b-da64-441e-a2be-2c00d04d28ab_isrForCrosstalkSources_1031732_17.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/a309025b-da64-441e-a2be-2c00d04d28ab_isrForCrosstalkSources_1031732_17.stderr parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b76fe8049d8e7bea1e1d9858348db20 with result from task 983 parsl.dataflow.dflow DEBUG: Task 983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/73805eb4-e0a3-4681-905f-4d52e844f866_isrForCrosstalkSources_1031700_33.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/73805eb4-e0a3-4681-905f-4d52e844f866_isrForCrosstalkSources_1031700_33.stderr parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce027fc6f842501cfb350b495ce78867 with result from task 939 parsl.dataflow.dflow DEBUG: Task 939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/33b2f4ef-6f99-45e8-8e05-c8957e01fc7a_isrForCrosstalkSources_1031709_42.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/33b2f4ef-6f99-45e8-8e05-c8957e01fc7a_isrForCrosstalkSources_1031709_42.stderr parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 105d37ea355a35635464efba606dbf2b with result from task 994 parsl.dataflow.dflow DEBUG: Task 994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/dc41d75c-ae66-46b4-8da4-c88c347c36d6_isrForCrosstalkSources_1031712_8.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/dc41d75c-ae66-46b4-8da4-c88c347c36d6_isrForCrosstalkSources_1031712_8.stderr parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d3f04f73e4b91fcf38fa055fefc63bc with result from task 988 parsl.dataflow.dflow DEBUG: Task 988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/aa7b7896-72e3-4eae-a71e-0d47619c36fb_isrForCrosstalkSources_1031712_39.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/aa7b7896-72e3-4eae-a71e-0d47619c36fb_isrForCrosstalkSources_1031712_39.stderr parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0674b049706b4fccea97fd67855d82f0 with result from task 1108 parsl.dataflow.dflow DEBUG: Task 1108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e1dd2b03-e249-448b-b446-c3fd14c8c98c_isrForCrosstalkSources_1031731_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e1dd2b03-e249-448b-b446-c3fd14c8c98c_isrForCrosstalkSources_1031731_13.stderr parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f86752dab14a2b71967f0f2a98ec5ad8 with result from task 1059 parsl.dataflow.dflow DEBUG: Task 1059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/c16fff75-686f-4ff5-9103-aa6db593eda4_isrForCrosstalkSources_1031711_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/c16fff75-686f-4ff5-9103-aa6db593eda4_isrForCrosstalkSources_1031711_42.stderr 2024-03-17 04:05:48 proc_lsst.multi:170 [INFO] [multi] job_id 17148075 proc_lsst.multi INFO: [multi] job_id 17148075 2024-03-17 04:05:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 74 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 74 2024-03-17 04:05:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148075 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148075 parsl.executors.status_handling DEBUG: Launched block 129 on executor multi with job ID 17148075 parsl.executors.status_handling INFO: Allocated block ID 130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-130 2024-03-17 04:05:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 2024-03-17 04:05:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 74 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 74 2024-03-17 04:05:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e24a5c030cb0236ccce57bf5a02d89e3 with result from task 960 parsl.dataflow.dflow DEBUG: Task 960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f075f619-1266-48f7-a512-cabcc679c417_isrForCrosstalkSources_1031720_2.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f075f619-1266-48f7-a512-cabcc679c417_isrForCrosstalkSources_1031720_2.stderr parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31304960c0506db8cf9579105eaee9fd with result from task 993 parsl.dataflow.dflow DEBUG: Task 993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/b2335663-c8aa-47aa-b96b-da8708296f2a_isrForCrosstalkSources_1031700_45.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/b2335663-c8aa-47aa-b96b-da8708296f2a_isrForCrosstalkSources_1031700_45.stderr parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4e75801575139cadb8bea3cd59ca8dc with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6c4344cd-282d-4909-87de-f9b3050fa2a1_isrForCrosstalkSources_1031721_53.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6c4344cd-282d-4909-87de-f9b3050fa2a1_isrForCrosstalkSources_1031721_53.stderr parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5483a4e99bf45f8ec9bb7d5857d2c45 with result from task 1003 parsl.dataflow.dflow DEBUG: Task 1003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/478f88f1-e8b8-409b-823a-95d70337e7eb_isrForCrosstalkSources_1031723_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/478f88f1-e8b8-409b-823a-95d70337e7eb_isrForCrosstalkSources_1031723_34.stderr parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e444ffd2aaba9a11bd186eba731f73c with result from task 1104 parsl.dataflow.dflow DEBUG: Task 1104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ecc7a798-e303-4933-9c6a-cb7694d7a437_isrForCrosstalkSources_1031738_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ecc7a798-e303-4933-9c6a-cb7694d7a437_isrForCrosstalkSources_1031738_32.stderr parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89674ff2b10d62d401b6ba9736586f32 with result from task 1014 parsl.dataflow.dflow DEBUG: Task 1014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bb74893c-7394-414d-9ab0-785c06a1a25c_isrForCrosstalkSources_1031729_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bb74893c-7394-414d-9ab0-785c06a1a25c_isrForCrosstalkSources_1031729_34.stderr parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22ebbf1653230a9b69b21c4a14e0b987 with result from task 1074 parsl.dataflow.dflow DEBUG: Task 1074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e14213dd-9ea6-4285-bb66-e8a18cc4f94e_isrForCrosstalkSources_1031738_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e14213dd-9ea6-4285-bb66-e8a18cc4f94e_isrForCrosstalkSources_1031738_9.stderr parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 371372799656ff50457ce9677700e882 with result from task 1094 parsl.dataflow.dflow DEBUG: Task 1094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/0e8f1d21-584b-4ff2-b2be-991832986e63_isrForCrosstalkSources_1031702_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/0e8f1d21-584b-4ff2-b2be-991832986e63_isrForCrosstalkSources_1031702_46.stderr parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1db19111a79e0c9aafdd10d8d0226888 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/44db1db4-b147-4e60-807f-67602d36767a_isrForCrosstalkSources_1031727_23.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/44db1db4-b147-4e60-807f-67602d36767a_isrForCrosstalkSources_1031727_23.stderr parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea0acda142c8178abd255f494cfc2367 with result from task 1061 parsl.dataflow.dflow DEBUG: Task 1061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/4fac5e8b-7af2-485f-b38b-9e908effaa79_isrForCrosstalkSources_1031707_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/4fac5e8b-7af2-485f-b38b-9e908effaa79_isrForCrosstalkSources_1031707_45.stderr parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c574277e4f1120c7e4b8d3a1263c2dc with result from task 1068 parsl.dataflow.dflow DEBUG: Task 1068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/0a8445a2-468f-4bab-a2cb-1201fe593041_isrForCrosstalkSources_1031716_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/0a8445a2-468f-4bab-a2cb-1201fe593041_isrForCrosstalkSources_1031716_15.stderr 2024-03-17 04:05:48 proc_lsst.multi:170 [INFO] [multi] job_id 17148076 proc_lsst.multi INFO: [multi] job_id 17148076 2024-03-17 04:05:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 75 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 75 2024-03-17 04:05:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148076 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148076 parsl.executors.status_handling DEBUG: Launched block 130 on executor multi with job ID 17148076 parsl.executors.status_handling INFO: Allocated block ID 131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-131 2024-03-17 04:05:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 2024-03-17 04:05:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 75 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 75 2024-03-17 04:05:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ee479a8fe50f03a0214d85984e6cc3c with result from task 1046 parsl.dataflow.dflow DEBUG: Task 1046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7e6932e8-9188-449f-b947-c641e38d4845_isrForCrosstalkSources_1031729_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7e6932e8-9188-449f-b947-c641e38d4845_isrForCrosstalkSources_1031729_29.stderr parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a748cc07b88404148ba8e678a61d3b12 with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f2dbaa67-58da-4b7e-9e15-71ab12506da0_isrForCrosstalkSources_1031721_17.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f2dbaa67-58da-4b7e-9e15-71ab12506da0_isrForCrosstalkSources_1031721_17.stderr parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a9bf99bd75ac0bb77be822747126e3e with result from task 1050 parsl.dataflow.dflow DEBUG: Task 1050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d62877a0-4555-4360-9f17-b90d5c675237_isrForCrosstalkSources_1031714_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d62877a0-4555-4360-9f17-b90d5c675237_isrForCrosstalkSources_1031714_62.stderr parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 617b9f16a532d963036c2d12ac567f2b with result from task 961 parsl.dataflow.dflow DEBUG: Task 961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2f069e2b-9335-465a-a28f-f8d8a4dbf935_isrForCrosstalkSources_1031703_29.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2f069e2b-9335-465a-a28f-f8d8a4dbf935_isrForCrosstalkSources_1031703_29.stderr parsl.dataflow.dflow INFO: Task 992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a36faa83144c09908e08cb82ebfdba0 with result from task 992 parsl.dataflow.dflow DEBUG: Task 992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/78d7105f-ff93-4999-a6fa-b7879a1d3fbd_isrForCrosstalkSources_1031716_13.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/78d7105f-ff93-4999-a6fa-b7879a1d3fbd_isrForCrosstalkSources_1031716_13.stderr parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c14441a8fa25be68868f8f4ad9156d4 with result from task 1031 parsl.dataflow.dflow DEBUG: Task 1031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/0c8274e9-3fd0-4602-aed8-687519b63d1d_isrForCrosstalkSources_1031738_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/0c8274e9-3fd0-4602-aed8-687519b63d1d_isrForCrosstalkSources_1031738_62.stderr parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5427a1d4f1687f2fbb60918b74750e62 with result from task 1083 parsl.dataflow.dflow DEBUG: Task 1083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d2017012-fe24-43b8-bc78-8f9fb7df6fcd_isrForCrosstalkSources_1031722_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d2017012-fe24-43b8-bc78-8f9fb7df6fcd_isrForCrosstalkSources_1031722_6.stderr parsl.dataflow.dflow INFO: Task 1134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc10bcc00827a5e455008a60c345a808 with result from task 1134 parsl.dataflow.dflow DEBUG: Task 1134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/124402a8-ec66-468c-9983-4ab26cbaba12_isrForCrosstalkSources_1031733_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/124402a8-ec66-468c-9983-4ab26cbaba12_isrForCrosstalkSources_1031733_6.stderr parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aad107ca4fd231c93ae69ab7d4b2fea0 with result from task 987 parsl.dataflow.dflow DEBUG: Task 987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/9adcaf62-5442-4ebb-8299-58ab3bc4b694_isrForCrosstalkSources_1031727_56.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/9adcaf62-5442-4ebb-8299-58ab3bc4b694_isrForCrosstalkSources_1031727_56.stderr parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 950cf36b4a26d714cd8675edf6da78f3 with result from task 931 parsl.dataflow.dflow DEBUG: Task 931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/e9e8b19f-ec80-43ae-ac6f-efc80d7d8e0a_isrForCrosstalkSources_1031724_41.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/e9e8b19f-ec80-43ae-ac6f-efc80d7d8e0a_isrForCrosstalkSources_1031724_41.stderr 2024-03-17 04:05:49 proc_lsst.multi:170 [INFO] [multi] job_id 17148077 proc_lsst.multi INFO: [multi] job_id 17148077 2024-03-17 04:05:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 76 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 76 2024-03-17 04:05:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148077 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148077 parsl.executors.status_handling DEBUG: Launched block 131 on executor multi with job ID 17148077 parsl.executors.status_handling INFO: Allocated block ID 132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-132 2024-03-17 04:05:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 2024-03-17 04:05:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 76 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 76 2024-03-17 04:05:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4fbee516e0d8d597986fc9dc1f8a1a1 with result from task 1140 parsl.dataflow.dflow DEBUG: Task 1140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5f68dcc8-a0e5-43e5-b4fa-51819cfd4f44_isrForCrosstalkSources_1031730_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5f68dcc8-a0e5-43e5-b4fa-51819cfd4f44_isrForCrosstalkSources_1031730_15.stderr parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 957b986e909d9db07293b10b0361c9b4 with result from task 1095 parsl.dataflow.dflow DEBUG: Task 1095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5ef68d23-93f1-418c-93ae-8ca0313a8182_isrForCrosstalkSources_1031736_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5ef68d23-93f1-418c-93ae-8ca0313a8182_isrForCrosstalkSources_1031736_16.stderr parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ea4f7647e189c5cddb57b6aa7b8dd11 with result from task 991 parsl.dataflow.dflow DEBUG: Task 991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/38a5ab9c-33b3-4bb1-8c70-bbc97ddecf09_isrForCrosstalkSources_1031706_14.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/38a5ab9c-33b3-4bb1-8c70-bbc97ddecf09_isrForCrosstalkSources_1031706_14.stderr parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 985f0a5342f21cea54914b86b15404d0 with result from task 1048 parsl.dataflow.dflow DEBUG: Task 1048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/cd0654dd-240a-4643-99a2-b7fb705f3b1e_isrForCrosstalkSources_1031709_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/cd0654dd-240a-4643-99a2-b7fb705f3b1e_isrForCrosstalkSources_1031709_14.stderr parsl.dataflow.dflow INFO: Task 1106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d5c95398d4f7a9c5c4b16f55dce74b6 with result from task 1106 parsl.dataflow.dflow DEBUG: Task 1106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7c3283f2-05c7-4d18-8b5b-35521d71053c_isrForCrosstalkSources_1031725_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7c3283f2-05c7-4d18-8b5b-35521d71053c_isrForCrosstalkSources_1031725_5.stderr parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 156aaebbe58d161ab4e1193514c374b5 with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/32861743-6122-4355-966e-8b0d4e33904f_isrForCrosstalkSources_1031705_49.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/32861743-6122-4355-966e-8b0d4e33904f_isrForCrosstalkSources_1031705_49.stderr parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 650e7c710fe4b9c867dbdbc2440ae3f5 with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/74369388-da3e-4f34-9f65-41852c4df044_isrForCrosstalkSources_1031729_27.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/74369388-da3e-4f34-9f65-41852c4df044_isrForCrosstalkSources_1031729_27.stderr parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80e1dff7b873a31cff5f051484e38333 with result from task 1077 parsl.dataflow.dflow DEBUG: Task 1077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a51c86d9-21ae-48e0-8c7e-bb01cbd38692_isrForCrosstalkSources_1031701_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a51c86d9-21ae-48e0-8c7e-bb01cbd38692_isrForCrosstalkSources_1031701_12.stderr parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f27902c1d41a0154fcf6f52b0ecea14c with result from task 1082 parsl.dataflow.dflow DEBUG: Task 1082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/25cf3cf5-5cd1-4639-ab00-1f6131841bef_isrForCrosstalkSources_1031713_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/25cf3cf5-5cd1-4639-ab00-1f6131841bef_isrForCrosstalkSources_1031713_33.stderr parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dc9cc9d9ea9d5bd4cccee58e6f15995 with result from task 803 parsl.dataflow.dflow DEBUG: Task 803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0556f597-0192-4984-bef6-4e26c8c6fde3_isrForCrosstalkSources_1031724_27.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0556f597-0192-4984-bef6-4e26c8c6fde3_isrForCrosstalkSources_1031724_27.stderr parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25f22b95ba660dfb30d5e14b37601654 with result from task 1079 parsl.dataflow.dflow DEBUG: Task 1079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/eaf72f9d-9dc1-473b-8dee-00b63a29d3a9_isrForCrosstalkSources_1031700_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/eaf72f9d-9dc1-473b-8dee-00b63a29d3a9_isrForCrosstalkSources_1031700_21.stderr parsl.dataflow.dflow INFO: Task 1128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7997808485b5ab0dc1ae4c26949e47c0 with result from task 1128 parsl.dataflow.dflow DEBUG: Task 1128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/09626f37-a8e4-4b54-9a1d-11cb036473c4_isrForCrosstalkSources_1031724_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/09626f37-a8e4-4b54-9a1d-11cb036473c4_isrForCrosstalkSources_1031724_9.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd054e2e33659da454b00469150d953e with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/3944cb57-6744-48fe-be5d-59feaca4e22d_isrForCrosstalkSources_1031715_34.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/3944cb57-6744-48fe-be5d-59feaca4e22d_isrForCrosstalkSources_1031715_34.stderr parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77bf099e75700ae99bb7275f85f689a5 with result from task 989 parsl.dataflow.dflow DEBUG: Task 989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/8a26e547-6fe3-45ff-8304-6d08dd5d7c84_isrForCrosstalkSources_1031705_45.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/8a26e547-6fe3-45ff-8304-6d08dd5d7c84_isrForCrosstalkSources_1031705_45.stderr parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b73e0430d82113af3458883430fa41b with result from task 1053 parsl.dataflow.dflow DEBUG: Task 1053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/944b3532-f330-44ac-9b41-388952d287f0_isrForCrosstalkSources_1031722_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/944b3532-f330-44ac-9b41-388952d287f0_isrForCrosstalkSources_1031722_22.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 145d7a33ee49f0a3e21c28d688c2b008 with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/34eac1bb-0834-442b-ab32-a105054dc8a8_isrForCrosstalkSources_1031720_34.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/34eac1bb-0834-442b-ab32-a105054dc8a8_isrForCrosstalkSources_1031720_34.stderr parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec8c811a571429bca20aaa4cd985acf3 with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ea812b46-975d-416f-a48f-0e3345807ff1_isrForCrosstalkSources_1031711_31.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ea812b46-975d-416f-a48f-0e3345807ff1_isrForCrosstalkSources_1031711_31.stderr parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8def4b294bae98cb19f3ccd3b460788d with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/541d2996-81ad-4280-9ed4-f17a567f7606_isrForCrosstalkSources_1031712_13.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/541d2996-81ad-4280-9ed4-f17a567f7606_isrForCrosstalkSources_1031712_13.stderr parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d84c30d878ac0e031c5b6e2e38fc5ce2 with result from task 1081 parsl.dataflow.dflow DEBUG: Task 1081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7b9bfda0-3b05-4d42-b985-e682c94da391_isrForCrosstalkSources_1031725_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7b9bfda0-3b05-4d42-b985-e682c94da391_isrForCrosstalkSources_1031725_19.stderr parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 315b87538743c5d1b1e0b4916255146e with result from task 999 parsl.dataflow.dflow DEBUG: Task 999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/dc10ffe5-073e-4189-b2f7-f0eb5b96e036_isrForCrosstalkSources_1031737_38.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/dc10ffe5-073e-4189-b2f7-f0eb5b96e036_isrForCrosstalkSources_1031737_38.stderr parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bc363971e2c01b051ca5a85c83a8ba6 with result from task 1004 parsl.dataflow.dflow DEBUG: Task 1004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/1690753d-9bac-4268-a70c-16b8ae3bd993_isrForCrosstalkSources_1031737_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/1690753d-9bac-4268-a70c-16b8ae3bd993_isrForCrosstalkSources_1031737_9.stderr parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ada2d7c57898aed4d421d92192e4065b with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c84d6e2e-0e29-4bba-994d-a6b1ac088bf7_isrForCrosstalkSources_1031707_46.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c84d6e2e-0e29-4bba-994d-a6b1ac088bf7_isrForCrosstalkSources_1031707_46.stderr parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74574dd2e2e4309311491d4c1f43c35d with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/810abadd-e6e2-465a-8ea8-c6509b1ab6d5_isrForCrosstalkSources_1031722_5.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/810abadd-e6e2-465a-8ea8-c6509b1ab6d5_isrForCrosstalkSources_1031722_5.stderr 2024-03-17 04:05:50 proc_lsst.multi:170 [INFO] [multi] job_id 17148078 proc_lsst.multi INFO: [multi] job_id 17148078 2024-03-17 04:05:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 77 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 77 2024-03-17 04:05:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148078 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148078 parsl.executors.status_handling DEBUG: Launched block 132 on executor multi with job ID 17148078 parsl.executors.status_handling INFO: Allocated block ID 133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-133 2024-03-17 04:05:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 2024-03-17 04:05:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 77 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 77 2024-03-17 04:05:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization DEBUG: Storing app cache entry 40ab67faf5a1d14620ea546b6dd9d2aa with result from task 1093 parsl.dataflow.dflow DEBUG: Task 1093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2c939247-974f-4d5b-9d27-8b340f5ca313_isrForCrosstalkSources_1031724_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2c939247-974f-4d5b-9d27-8b340f5ca313_isrForCrosstalkSources_1031724_43.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccfd50ed08211b6da22d8116e9099fbe with result from task 1001 parsl.dataflow.dflow DEBUG: Task 1001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/562efe41-c410-4ae5-93b4-b306a512f584_isrForCrosstalkSources_1031735_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/562efe41-c410-4ae5-93b4-b306a512f584_isrForCrosstalkSources_1031735_46.stderr parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc0ba5f3bc53584087cf12b7a81ca1b8 with result from task 1043 parsl.dataflow.dflow DEBUG: Task 1043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/caef9003-7236-41f8-9f68-4de6cfd65d6c_isrForCrosstalkSources_1031719_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/caef9003-7236-41f8-9f68-4de6cfd65d6c_isrForCrosstalkSources_1031719_11.stderr parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caf122574eb0a4e6f863b0fb6f9a1cff with result from task 1092 parsl.dataflow.dflow DEBUG: Task 1092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/4900ac4e-eea4-4f22-baf8-34e37503d4c5_isrForCrosstalkSources_1031713_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/4900ac4e-eea4-4f22-baf8-34e37503d4c5_isrForCrosstalkSources_1031713_37.stderr parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0eb5625eb1cdee1124e43b8d08b99f7 with result from task 1080 parsl.dataflow.dflow DEBUG: Task 1080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/653ca142-6ee9-4e19-8e63-6dbdab2f7cde_isrForCrosstalkSources_1031712_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/653ca142-6ee9-4e19-8e63-6dbdab2f7cde_isrForCrosstalkSources_1031712_48.stderr parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 829a3fd992e808962e3187267b94298c with result from task 1062 parsl.dataflow.dflow DEBUG: Task 1062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f04c0d2a-dde4-4a20-91f9-1239103e1c4a_isrForCrosstalkSources_1031723_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f04c0d2a-dde4-4a20-91f9-1239103e1c4a_isrForCrosstalkSources_1031723_30.stderr parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35e6e5bcc9b52d1664b0d1167725f543 with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/fd30ee5b-aa09-4aa5-8cc5-27631ea2b425_isrForCrosstalkSources_1031724_1.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/fd30ee5b-aa09-4aa5-8cc5-27631ea2b425_isrForCrosstalkSources_1031724_1.stderr parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7db72728f90de5835cce0162e9176dc2 with result from task 1058 parsl.dataflow.dflow DEBUG: Task 1058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/1a4b66c5-3d99-48a1-bb82-0a6531c5686c_isrForCrosstalkSources_1031718_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/1a4b66c5-3d99-48a1-bb82-0a6531c5686c_isrForCrosstalkSources_1031718_23.stderr parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee65648a2cffca0de1e47bdf5f58279e with result from task 964 parsl.dataflow.dflow DEBUG: Task 964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/891e1f80-6e1d-40a2-a092-1d744333e6ff_isrForCrosstalkSources_1031734_44.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/891e1f80-6e1d-40a2-a092-1d744333e6ff_isrForCrosstalkSources_1031734_44.stderr 2024-03-17 04:05:50 proc_lsst.multi:170 [INFO] [multi] job_id 17148079 proc_lsst.multi INFO: [multi] job_id 17148079 2024-03-17 04:05:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 78 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 78 2024-03-17 04:05:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148079 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148079 parsl.executors.status_handling DEBUG: Launched block 133 on executor multi with job ID 17148079 parsl.executors.status_handling INFO: Allocated block ID 134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-134 2024-03-17 04:05:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 2024-03-17 04:05:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 78 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 78 2024-03-17 04:05:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1808d5f7c4894f7bac21656fa3d3e674 with result from task 1056 parsl.dataflow.dflow DEBUG: Task 1056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/62793ff0-afbf-4a72-99c4-6045961e79d0_isrForCrosstalkSources_1031725_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/62793ff0-afbf-4a72-99c4-6045961e79d0_isrForCrosstalkSources_1031725_21.stderr parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e2b11a290fc5e01c1eea79b43f717f1 with result from task 1107 parsl.dataflow.dflow DEBUG: Task 1107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/baf602bf-5009-47ff-bf79-3bdc9c100df5_isrForCrosstalkSources_1031700_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/baf602bf-5009-47ff-bf79-3bdc9c100df5_isrForCrosstalkSources_1031700_41.stderr parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1fdc9c46fa9de74bd79eb3be8a19476 with result from task 1049 parsl.dataflow.dflow DEBUG: Task 1049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/740d1dfb-0c56-4a0f-ba3b-3fdca58c1cd5_isrForCrosstalkSources_1031738_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/740d1dfb-0c56-4a0f-ba3b-3fdca58c1cd5_isrForCrosstalkSources_1031738_39.stderr parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c79ee4f72db2ddf9b72155da93820ca0 with result from task 1029 parsl.dataflow.dflow DEBUG: Task 1029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7efc6b0e-2ce8-45d6-aacc-c951b441fe59_isrForCrosstalkSources_1031704_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7efc6b0e-2ce8-45d6-aacc-c951b441fe59_isrForCrosstalkSources_1031704_10.stderr parsl.dataflow.dflow INFO: Task 1141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a313b87ce5488f86259f39ce22fdf91 with result from task 1141 parsl.dataflow.dflow DEBUG: Task 1141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/bd72e86e-f4d3-4036-909a-a58075eeccaa_isrForCrosstalkSources_1031700_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/bd72e86e-f4d3-4036-909a-a58075eeccaa_isrForCrosstalkSources_1031700_49.stderr parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e17dda319bf797b23cea267522505c8 with result from task 1067 parsl.dataflow.dflow DEBUG: Task 1067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fb2d0e30-8b28-4b11-9454-df57545534cb_isrForCrosstalkSources_1031732_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fb2d0e30-8b28-4b11-9454-df57545534cb_isrForCrosstalkSources_1031732_12.stderr parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 139f4c2bb5025e9488f7f95245659e0c with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/f65afd00-5355-404d-88d1-6c2cef3206dd_isrForCrosstalkSources_1031712_5.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/f65afd00-5355-404d-88d1-6c2cef3206dd_isrForCrosstalkSources_1031712_5.stderr parsl.dataflow.dflow INFO: Task 1138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a723e92c4968860843735529a8803508 with result from task 1138 parsl.dataflow.dflow DEBUG: Task 1138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/917aa082-ef62-4101-a4e6-4e40a353a654_isrForCrosstalkSources_1031711_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/917aa082-ef62-4101-a4e6-4e40a353a654_isrForCrosstalkSources_1031711_5.stderr parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33b72f64c0cd3b8fe0fda54eea83025d with result from task 997 parsl.dataflow.dflow DEBUG: Task 997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/9f07a476-3127-47ec-a3e0-fa77740d4d2a_isrForCrosstalkSources_1031715_40.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/9f07a476-3127-47ec-a3e0-fa77740d4d2a_isrForCrosstalkSources_1031715_40.stderr parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0d097faf6174f3e2b4dd5346f07359f with result from task 1021 parsl.dataflow.dflow DEBUG: Task 1021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f45af25f-77fe-4996-86cd-65224127a010_isrForCrosstalkSources_1031700_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/f45af25f-77fe-4996-86cd-65224127a010_isrForCrosstalkSources_1031700_32.stderr parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae078d042eb5c2f21358aad3cec60fec with result from task 1047 parsl.dataflow.dflow DEBUG: Task 1047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/57efe522-1e5c-496b-a379-355e8dab41f8_isrForCrosstalkSources_1031699_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/57efe522-1e5c-496b-a379-355e8dab41f8_isrForCrosstalkSources_1031699_11.stderr parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e095c991ebf2ca5fbf98eaf5b0fb8a74 with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/28313962-4c12-4ec7-a674-bf49be8cd37f_isrForCrosstalkSources_1031727_61.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/28313962-4c12-4ec7-a674-bf49be8cd37f_isrForCrosstalkSources_1031727_61.stderr parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38c2420893a0e6ec4eeb1276d30a7760 with result from task 1026 parsl.dataflow.dflow DEBUG: Task 1026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/41beba85-719b-45b4-89ba-ddcfe84cbf6e_isrForCrosstalkSources_1031719_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/41beba85-719b-45b4-89ba-ddcfe84cbf6e_isrForCrosstalkSources_1031719_45.stderr 2024-03-17 04:05:51 proc_lsst.multi:170 [INFO] [multi] job_id 17148080 proc_lsst.multi INFO: [multi] job_id 17148080 2024-03-17 04:05:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 79 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 79 2024-03-17 04:05:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148080 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148080 parsl.executors.status_handling DEBUG: Launched block 134 on executor multi with job ID 17148080 parsl.executors.status_handling INFO: Allocated block ID 135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-135 2024-03-17 04:05:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 2024-03-17 04:05:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 79 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 79 2024-03-17 04:05:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3530210d17ff1a2542094a1f735a9bee with result from task 1022 parsl.dataflow.dflow DEBUG: Task 1022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c6401ebf-03ae-4d1a-8af6-eba01a10440d_isrForCrosstalkSources_1031732_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c6401ebf-03ae-4d1a-8af6-eba01a10440d_isrForCrosstalkSources_1031732_19.stderr parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3efc665e80e0e179b51c9061d0422fc6 with result from task 1063 parsl.dataflow.dflow DEBUG: Task 1063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e71de4b6-b3df-4f2e-b392-e4f82ffa73dd_isrForCrosstalkSources_1031725_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e71de4b6-b3df-4f2e-b392-e4f82ffa73dd_isrForCrosstalkSources_1031725_38.stderr parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccd8a96d8da8a49fbd313c40ecd6dc41 with result from task 1054 parsl.dataflow.dflow DEBUG: Task 1054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d1db1de1-3111-401f-83ba-53765d85cc49_isrForCrosstalkSources_1031727_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/d1db1de1-3111-401f-83ba-53765d85cc49_isrForCrosstalkSources_1031727_2.stderr parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c950a1de8f776cf738fd0847d51d8e82 with result from task 932 parsl.dataflow.dflow DEBUG: Task 932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/aedc4585-34ca-447a-80b0-917eea51e7bf_isrForCrosstalkSources_1031700_54.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/aedc4585-34ca-447a-80b0-917eea51e7bf_isrForCrosstalkSources_1031700_54.stderr parsl.dataflow.dflow INFO: Task 1198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8edc29a3392ee7701ba7de18e1159136 with result from task 1198 parsl.dataflow.dflow DEBUG: Task 1198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0d8f1e8c-3c97-437e-9205-3dd24b3db345_isrForCrosstalkSources_1031711_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0d8f1e8c-3c97-437e-9205-3dd24b3db345_isrForCrosstalkSources_1031711_3.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff6b9073af2ddf16ea3eaebc9caedf31 with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4c613071-5920-455a-8633-a33dd2e19e7a_isrForCrosstalkSources_1031738_47.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4c613071-5920-455a-8633-a33dd2e19e7a_isrForCrosstalkSources_1031738_47.stderr parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b557c72b15e27b21c2074ba180b4775 with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/f94cf3f3-2da5-4243-8189-b108329bf71b_isrForCrosstalkSources_1031734_6.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/f94cf3f3-2da5-4243-8189-b108329bf71b_isrForCrosstalkSources_1031734_6.stderr parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c70a1ff1fac66dcfb48395c2d5035871 with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4bfbf50c-3afe-4b75-a1c9-0d456c0ccdb0_isrForCrosstalkSources_1031738_45.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4bfbf50c-3afe-4b75-a1c9-0d456c0ccdb0_isrForCrosstalkSources_1031738_45.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 193189ca7f6ac9a50145c5fe29f947d4 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/7a19b6af-0dc2-4413-b452-d1cc718be8b7_isrForCrosstalkSources_1031717_2.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/7a19b6af-0dc2-4413-b452-d1cc718be8b7_isrForCrosstalkSources_1031717_2.stderr parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00007b751c1125a53f49a245afdc5b0d with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/35a4d7e3-b0f2-4979-8d35-969ee3295d80_isrForCrosstalkSources_1031713_24.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/35a4d7e3-b0f2-4979-8d35-969ee3295d80_isrForCrosstalkSources_1031713_24.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91e7a26a3c3b833321f16ebeea5811bb with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/6e21091e-ef8f-465d-8adf-fbca7035563d_isrForCrosstalkSources_1031714_46.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/6e21091e-ef8f-465d-8adf-fbca7035563d_isrForCrosstalkSources_1031714_46.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c2fb487a46dc6fff6d551efa6bbd2ae with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/25f58b1f-aaad-4c30-b5f3-36c9b1b8a7b3_isrForCrosstalkSources_1031705_56.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/25f58b1f-aaad-4c30-b5f3-36c9b1b8a7b3_isrForCrosstalkSources_1031705_56.stderr parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d1086196f5540b438231005306db59f with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/659fbf53-7aac-4687-a844-6ac2d48034f0_isrForCrosstalkSources_1031712_49.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/659fbf53-7aac-4687-a844-6ac2d48034f0_isrForCrosstalkSources_1031712_49.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edce8d0c4ac22fd93760c4bdfcdb856d with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/35d662cd-d72a-4689-8593-687bb001d504_isrForCrosstalkSources_1031708_43.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/35d662cd-d72a-4689-8593-687bb001d504_isrForCrosstalkSources_1031708_43.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2b374cb204cb45ad124a9a1c5d08f11 with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/729a844a-7d7f-4f05-95cb-51aeacf0910a_isrForCrosstalkSources_1031724_26.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/729a844a-7d7f-4f05-95cb-51aeacf0910a_isrForCrosstalkSources_1031724_26.stderr parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96591160d953026645a0f8c5a33d0521 with result from task 745 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6e225857-957b-4133-9064-30858df27fa6_isrForCrosstalkSources_1031721_6.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6e225857-957b-4133-9064-30858df27fa6_isrForCrosstalkSources_1031721_6.stderr parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a5bf4a137042c2708bc8093a25f6aa6 with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/34bb3a6f-aafb-4301-aa34-1ea105d4315c_isrForCrosstalkSources_1031716_36.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/34bb3a6f-aafb-4301-aa34-1ea105d4315c_isrForCrosstalkSources_1031716_36.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e529170658a473762b5408c8d5b97d3d with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/1893b7eb-f761-4c0c-988b-ac766c9cc90c_isrForCrosstalkSources_1031714_33.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/1893b7eb-f761-4c0c-988b-ac766c9cc90c_isrForCrosstalkSources_1031714_33.stderr parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a70af98cd3bc3fde53a6329ed10e29d with result from task 969 parsl.dataflow.dflow DEBUG: Task 969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/33bf5146-4d7f-4eed-9c23-965f98a414e7_isrForCrosstalkSources_1031721_35.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/33bf5146-4d7f-4eed-9c23-965f98a414e7_isrForCrosstalkSources_1031721_35.stderr parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10f65fd84706d5e5324dbe689164689a with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e8cced28-1905-4566-b185-be3f05185d90_isrForCrosstalkSources_1031728_3.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e8cced28-1905-4566-b185-be3f05185d90_isrForCrosstalkSources_1031728_3.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45ccd9c87ff74cf19bff360548ed564e with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9c2f6284-5b13-4e00-906e-ec0a9fb658d1_isrForCrosstalkSources_1031733_11.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9c2f6284-5b13-4e00-906e-ec0a9fb658d1_isrForCrosstalkSources_1031733_11.stderr parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f666a0245cea32f189e3452c1470e18 with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/114caae0-cb3d-4203-bd24-6b41faaff8de_isrForCrosstalkSources_1031735_35.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/114caae0-cb3d-4203-bd24-6b41faaff8de_isrForCrosstalkSources_1031735_35.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8358943673a368e1bb2cd00ef371def with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/f7c6eb6d-9da5-4c83-a1ac-7015e03f64f2_isrForCrosstalkSources_1031728_37.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/f7c6eb6d-9da5-4c83-a1ac-7015e03f64f2_isrForCrosstalkSources_1031728_37.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3acb2f27f6c76501017f1c1682f80185 with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/54578a5c-d2b1-4a52-97eb-d1efbe9b7602_isrForCrosstalkSources_1031713_34.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/54578a5c-d2b1-4a52-97eb-d1efbe9b7602_isrForCrosstalkSources_1031713_34.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff3f0b0810115664b1156dca48aed021 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5d026d10-8623-4a23-8496-47bdfb573774_isrForCrosstalkSources_1031705_8.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5d026d10-8623-4a23-8496-47bdfb573774_isrForCrosstalkSources_1031705_8.stderr parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1798addc09879b429926baaebd9c6a6c with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6bb6ffe4-250b-4290-a588-d6ac6bdd7418_isrForCrosstalkSources_1031716_28.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6bb6ffe4-250b-4290-a588-d6ac6bdd7418_isrForCrosstalkSources_1031716_28.stderr parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0784cd89c41b5ac94ebcddb04003bf06 with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/12f1faaf-098d-476c-a625-1129b0b1ed45_isrForCrosstalkSources_1031730_41.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/12f1faaf-098d-476c-a625-1129b0b1ed45_isrForCrosstalkSources_1031730_41.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c90552476606c3fb4147dfbd9718f110 with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/22e50df8-8715-40e1-9a83-dfbb7f2101b3_isrForCrosstalkSources_1031735_33.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/22e50df8-8715-40e1-9a83-dfbb7f2101b3_isrForCrosstalkSources_1031735_33.stderr parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c999b0fe92a671e32dbda2cc616ad3e5 with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/fc211aa4-2cda-4a8d-8561-19a6134ed460_isrForCrosstalkSources_1031701_61.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/fc211aa4-2cda-4a8d-8561-19a6134ed460_isrForCrosstalkSources_1031701_61.stderr parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d137c404f7fb18f0323b8fca6ed4aa65 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e78550d2-6541-4348-b6eb-d4932ee54e86_isrForCrosstalkSources_1031720_24.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e78550d2-6541-4348-b6eb-d4932ee54e86_isrForCrosstalkSources_1031720_24.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cb675b5f2b682bb771df2da16d60230 with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/81ea6377-b543-44a3-b943-58350c7063c5_isrForCrosstalkSources_1031715_56.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/81ea6377-b543-44a3-b943-58350c7063c5_isrForCrosstalkSources_1031715_56.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 735444fbf35eadcefd16fa4e6973a952 with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/2197f610-837c-4301-bd32-ac873eba30e7_isrForCrosstalkSources_1031699_44.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/2197f610-837c-4301-bd32-ac873eba30e7_isrForCrosstalkSources_1031699_44.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd3cfb17b115c2715d1a8daacb4791af with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/900ace74-a3e1-4a9a-882e-b3a846b83404_isrForCrosstalkSources_1031724_40.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/900ace74-a3e1-4a9a-882e-b3a846b83404_isrForCrosstalkSources_1031724_40.stderr parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a1349c20375c0a24873d66c892e4fec with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/25feb15b-3fae-48d4-adbe-a86b1d3801fc_isrForCrosstalkSources_1031714_47.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/25feb15b-3fae-48d4-adbe-a86b1d3801fc_isrForCrosstalkSources_1031714_47.stderr parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5510b5fa5d953252fda5bc8113175de with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cdd15980-4299-43c1-b0ae-d89e9476cb2a_isrForCrosstalkSources_1031727_41.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cdd15980-4299-43c1-b0ae-d89e9476cb2a_isrForCrosstalkSources_1031727_41.stderr parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cea3e50c086cfda8a29337f65e137903 with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a2164ffd-efda-464d-8ba1-4ef0b286d6cb_isrForCrosstalkSources_1031711_7.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a2164ffd-efda-464d-8ba1-4ef0b286d6cb_isrForCrosstalkSources_1031711_7.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed6c014e27901b3a830cb7f7f5257b86 with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8778c575-0da1-426b-8962-356d69165534_isrForCrosstalkSources_1031735_27.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/8778c575-0da1-426b-8962-356d69165534_isrForCrosstalkSources_1031735_27.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 236fbe3e4a2d9fef664f535d339aa27c with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/4506989d-3590-4180-84cc-f1a28afe7c96_isrForCrosstalkSources_1031722_57.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/4506989d-3590-4180-84cc-f1a28afe7c96_isrForCrosstalkSources_1031722_57.stderr parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa8982610c176b0bf13c4d573170db1 with result from task 1060 parsl.dataflow.dflow DEBUG: Task 1060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b5869329-1664-43d8-8531-c58446ca2e98_isrForCrosstalkSources_1031713_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b5869329-1664-43d8-8531-c58446ca2e98_isrForCrosstalkSources_1031713_21.stderr parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fc61610b67c9a8bf69b93bac0199b70 with result from task 1070 parsl.dataflow.dflow DEBUG: Task 1070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/b57a6446-2a82-4a69-9d9b-fd9e7803fa2b_isrForCrosstalkSources_1031735_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/b57a6446-2a82-4a69-9d9b-fd9e7803fa2b_isrForCrosstalkSources_1031735_36.stderr parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7a14540c9836a5bd9055d61a2f49e5a with result from task 1109 parsl.dataflow.dflow DEBUG: Task 1109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/213e916a-5542-4dfc-b003-5eca744dfe3f_isrForCrosstalkSources_1031736_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/213e916a-5542-4dfc-b003-5eca744dfe3f_isrForCrosstalkSources_1031736_21.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7271d2bdaeec185bba9c16abaa10b249 with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/03222f5e-c105-416d-a90a-1aae40913e9d_isrForCrosstalkSources_1031736_4.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/03222f5e-c105-416d-a90a-1aae40913e9d_isrForCrosstalkSources_1031736_4.stderr parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ca17e1540ac6a26aac956512bb48eaa with result from task 1064 parsl.dataflow.dflow DEBUG: Task 1064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/dcb4782d-cf49-422c-ae84-ee5c44ed9bf0_isrForCrosstalkSources_1031721_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/dcb4782d-cf49-422c-ae84-ee5c44ed9bf0_isrForCrosstalkSources_1031721_55.stderr parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a7d95e0261d776387dae890de019c71 with result from task 1071 parsl.dataflow.dflow DEBUG: Task 1071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4f18ac1c-6e4c-4fa2-b219-4ca9a058b07f_isrForCrosstalkSources_1031737_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4f18ac1c-6e4c-4fa2-b219-4ca9a058b07f_isrForCrosstalkSources_1031737_29.stderr parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e431e1de491620a1c5807372b674ee5 with result from task 1005 parsl.dataflow.dflow DEBUG: Task 1005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/84416d15-14ec-4fb1-813d-d528a6874f33_isrForCrosstalkSources_1031720_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/84416d15-14ec-4fb1-813d-d528a6874f33_isrForCrosstalkSources_1031720_6.stderr parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66c953155bb47356654e8e8b91dae457 with result from task 950 parsl.dataflow.dflow DEBUG: Task 950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/990f8c89-f19b-4bb7-a3d1-35a94173ae13_isrForCrosstalkSources_1031730_48.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/990f8c89-f19b-4bb7-a3d1-35a94173ae13_isrForCrosstalkSources_1031730_48.stderr parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c08b43d6e1086bc6d39c7e9e919ad37b with result from task 1007 parsl.dataflow.dflow DEBUG: Task 1007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0813cf5c-6429-45f0-8689-54ec82f75d68_isrForCrosstalkSources_1031720_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0813cf5c-6429-45f0-8689-54ec82f75d68_isrForCrosstalkSources_1031720_10.stderr parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c22fbdfde1f03f80b939c86ce1c21c62 with result from task 1012 parsl.dataflow.dflow DEBUG: Task 1012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/757b4c92-c43e-41ff-8999-17ad223c3cd6_isrForCrosstalkSources_1031705_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/757b4c92-c43e-41ff-8999-17ad223c3cd6_isrForCrosstalkSources_1031705_6.stderr parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1ce765a9900d8ee069bedca9de443c9 with result from task 996 parsl.dataflow.dflow DEBUG: Task 996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eb97c069-f42d-4092-84a7-78765bc627e6_isrForCrosstalkSources_1031719_12.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eb97c069-f42d-4092-84a7-78765bc627e6_isrForCrosstalkSources_1031719_12.stderr parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06449b31be24e7dc498dac4c8a9d0589 with result from task 1076 parsl.dataflow.dflow DEBUG: Task 1076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0992e56e-1be0-49ad-9087-2283f8396051_isrForCrosstalkSources_1031737_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0992e56e-1be0-49ad-9087-2283f8396051_isrForCrosstalkSources_1031737_44.stderr parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d003d40ea1aff436f7e07b53c7f9e421 with result from task 1008 parsl.dataflow.dflow DEBUG: Task 1008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/810c069f-842f-4783-97c1-1188943670b2_isrForCrosstalkSources_1031700_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/810c069f-842f-4783-97c1-1188943670b2_isrForCrosstalkSources_1031700_9.stderr parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 766bedc8c926b37484748064705075ef with result from task 1013 parsl.dataflow.dflow DEBUG: Task 1013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/55dbe9c5-4e49-43aa-84d5-8d6f2f880f8e_isrForCrosstalkSources_1031724_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/55dbe9c5-4e49-43aa-84d5-8d6f2f880f8e_isrForCrosstalkSources_1031724_16.stderr parsl.dataflow.dflow INFO: Task 1132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25acf4d1355bf2760fe01e3eede202e4 with result from task 1132 parsl.dataflow.dflow DEBUG: Task 1132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a4643fa5-b0a2-4596-8d78-2bcb748b819b_isrForCrosstalkSources_1031730_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a4643fa5-b0a2-4596-8d78-2bcb748b819b_isrForCrosstalkSources_1031730_61.stderr parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b26bdbf4ade0afe9c06d0f36e7c28cd with result from task 1024 parsl.dataflow.dflow DEBUG: Task 1024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/86dfa5e7-cf16-4e02-9599-91c26e3a9ce2_isrForCrosstalkSources_1031737_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/86dfa5e7-cf16-4e02-9599-91c26e3a9ce2_isrForCrosstalkSources_1031737_13.stderr parsl.dataflow.dflow INFO: Task 1142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79dcec1b8d1e6c09b660660099744f31 with result from task 1142 parsl.dataflow.dflow DEBUG: Task 1142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/2d1ff028-346c-4995-b1f9-835e040121da_isrForCrosstalkSources_1031717_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/2d1ff028-346c-4995-b1f9-835e040121da_isrForCrosstalkSources_1031717_55.stderr 2024-03-17 04:05:52 proc_lsst.multi:170 [INFO] [multi] job_id 17148081 proc_lsst.multi INFO: [multi] job_id 17148081 2024-03-17 04:05:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 80 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 80 2024-03-17 04:05:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148081 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148081 parsl.executors.status_handling DEBUG: Launched block 135 on executor multi with job ID 17148081 parsl.executors.status_handling INFO: Allocated block ID 136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-136 2024-03-17 04:05:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 2024-03-17 04:05:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 80 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 80 2024-03-17 04:05:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e4a33a2c97535bb120df6b638d9ebfa with result from task 998 parsl.dataflow.dflow DEBUG: Task 998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/96089947-a339-4bb9-8d16-4edbfaa82ec6_isrForCrosstalkSources_1031735_6.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/96089947-a339-4bb9-8d16-4edbfaa82ec6_isrForCrosstalkSources_1031735_6.stderr parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffa27af2d2d3e702a86351aba407fde1 with result from task 1044 parsl.dataflow.dflow DEBUG: Task 1044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/53382859-0a6f-4d96-9c5f-41d243e6409a_isrForCrosstalkSources_1031716_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/53382859-0a6f-4d96-9c5f-41d243e6409a_isrForCrosstalkSources_1031716_53.stderr parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d4a2d02e123b558b8a3a5337fa4dae7 with result from task 986 parsl.dataflow.dflow DEBUG: Task 986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b6eb666b-4282-4a70-b3fc-fe957fb918e1_isrForCrosstalkSources_1031721_2.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/b6eb666b-4282-4a70-b3fc-fe957fb918e1_isrForCrosstalkSources_1031721_2.stderr parsl.dataflow.dflow INFO: Task 1335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db7a5e3a16bcef90ac2ab10c3b9dcdd6 with result from task 1335 parsl.dataflow.dflow DEBUG: Task 1335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/9398fc64-df56-4fee-9da8-715f692c7171_isrForCrosstalkSources_1031728_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/9398fc64-df56-4fee-9da8-715f692c7171_isrForCrosstalkSources_1031728_25.stderr parsl.dataflow.dflow INFO: Task 1332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dc6c02979e7e29a06887f1d122a2dd1 with result from task 1332 parsl.dataflow.dflow DEBUG: Task 1332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/7449f93d-b28d-4281-88f6-77235a799cd5_isrForCrosstalkSources_1031734_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/7449f93d-b28d-4281-88f6-77235a799cd5_isrForCrosstalkSources_1031734_43.stderr parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c73ae4c2289c1b7465954b3e09612fe with result from task 984 parsl.dataflow.dflow DEBUG: Task 984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/eebe71aa-974c-4b22-817b-ff6ead4e9341_isrForCrosstalkSources_1031717_54.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/eebe71aa-974c-4b22-817b-ff6ead4e9341_isrForCrosstalkSources_1031717_54.stderr 2024-03-17 04:05:53 proc_lsst.multi:170 [INFO] [multi] job_id 17148082 proc_lsst.multi INFO: [multi] job_id 17148082 2024-03-17 04:05:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 81 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 81 2024-03-17 04:05:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148082 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148082 parsl.executors.status_handling DEBUG: Launched block 136 on executor multi with job ID 17148082 parsl.executors.status_handling INFO: Allocated block ID 137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-137 2024-03-17 04:05:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 2024-03-17 04:05:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 81 parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 328f83ce18e638be98ea0bc4d4776af4 with result from task 787 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 81 2024-03-17 04:05:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5931ecac-dee2-4a94-84be-20582c4cd51b_isrForCrosstalkSources_1031701_37.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5931ecac-dee2-4a94-84be-20582c4cd51b_isrForCrosstalkSources_1031701_37.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff65239a5d27ae3040cc66e6f7bfa166 with result from task 990 parsl.dataflow.dflow DEBUG: Task 990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/27eb1a40-da90-47cc-9ad3-7b5b3a9b6e35_isrForCrosstalkSources_1031707_52.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/27eb1a40-da90-47cc-9ad3-7b5b3a9b6e35_isrForCrosstalkSources_1031707_52.stderr parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eb6b87e0a95c3785fbc1e8e13b01475 with result from task 1010 parsl.dataflow.dflow DEBUG: Task 1010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/da714196-201e-430e-8c9e-ad38ea0d9446_isrForCrosstalkSources_1031707_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/da714196-201e-430e-8c9e-ad38ea0d9446_isrForCrosstalkSources_1031707_44.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4266d403c54938e05e79827d730d23c with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/71ac5674-01f4-4579-90bc-3bbd4fd13e47_isrForCrosstalkSources_1031705_51.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/71ac5674-01f4-4579-90bc-3bbd4fd13e47_isrForCrosstalkSources_1031705_51.stderr parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6db4d54cc7b20b34d4388e0bb0b1682 with result from task 977 parsl.dataflow.dflow DEBUG: Task 977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c0d80edd-f7a0-4652-b51f-a03f1a5d79b2_isrForCrosstalkSources_1031727_48.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c0d80edd-f7a0-4652-b51f-a03f1a5d79b2_isrForCrosstalkSources_1031727_48.stderr parsl.dataflow.dflow INFO: Task 1369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7d24d6d3719efc991d689770c6348ef with result from task 1369 parsl.dataflow.dflow DEBUG: Task 1369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/5b468d03-3351-4027-919d-d2e2d2d55020_isrForCrosstalkSources_1031715_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/5b468d03-3351-4027-919d-d2e2d2d55020_isrForCrosstalkSources_1031715_47.stderr parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e747f77f80b708d31c15233eb9635e49 with result from task 1075 parsl.dataflow.dflow DEBUG: Task 1075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/734ba3f2-e129-44c5-9b4d-60674cfae774_isrForCrosstalkSources_1031707_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/734ba3f2-e129-44c5-9b4d-60674cfae774_isrForCrosstalkSources_1031707_30.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d736337812431b488a2ace3d42e0782 with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e7f89c59-7bf3-48b8-9218-ce54249343cd_isrForCrosstalkSources_1031704_50.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e7f89c59-7bf3-48b8-9218-ce54249343cd_isrForCrosstalkSources_1031704_50.stderr parsl.dataflow.dflow INFO: Task 1334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 778424396274e60b6ba321c7f8b88626 with result from task 1334 parsl.dataflow.dflow DEBUG: Task 1334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0ad7cb09-c396-4989-b738-302ead4344b6_isrForCrosstalkSources_1031724_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/0ad7cb09-c396-4989-b738-302ead4344b6_isrForCrosstalkSources_1031724_56.stderr parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf1b3ba2b49745542ffa00eb572505be with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a30dab02-0844-4fc3-b256-e89f84352b38_isrForCrosstalkSources_1031731_41.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a30dab02-0844-4fc3-b256-e89f84352b38_isrForCrosstalkSources_1031731_41.stderr parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30a0e87cffc923a99b3ccb48dea62c81 with result from task 1023 parsl.dataflow.dflow DEBUG: Task 1023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/2f0fd5d4-b8e0-42c8-94c8-9baedd6cb6c6_isrForCrosstalkSources_1031718_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/2f0fd5d4-b8e0-42c8-94c8-9baedd6cb6c6_isrForCrosstalkSources_1031718_24.stderr 2024-03-17 04:05:54 proc_lsst.multi:170 [INFO] [multi] job_id 17148083 proc_lsst.multi INFO: [multi] job_id 17148083 2024-03-17 04:05:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 82 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 82 2024-03-17 04:05:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148083 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148083 parsl.executors.status_handling DEBUG: Launched block 137 on executor multi with job ID 17148083 parsl.executors.status_handling INFO: Allocated block ID 138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-138 2024-03-17 04:05:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 2024-03-17 04:05:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 82 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 82 2024-03-17 04:05:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb38d68fb6a464168831cf39333656e5 with result from task 1201 parsl.dataflow.dflow DEBUG: Task 1201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/b5492ec0-4493-4c7f-afe7-2de64b442b18_isrForCrosstalkSources_1031734_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/b5492ec0-4493-4c7f-afe7-2de64b442b18_isrForCrosstalkSources_1031734_53.stderr parsl.dataflow.dflow INFO: Task 1337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6128c9c13b986b52611c1dca7fc31b0a with result from task 1337 parsl.dataflow.dflow DEBUG: Task 1337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/73bb2342-ecf6-48c2-88d6-a76861d3aadd_isrForCrosstalkSources_1031722_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/73bb2342-ecf6-48c2-88d6-a76861d3aadd_isrForCrosstalkSources_1031722_48.stderr parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74193ab8435108e01088a43b347d4c21 with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/2cae5bd9-2ef2-4859-b685-f9ee51f9275e_isrForCrosstalkSources_1031721_9.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/2cae5bd9-2ef2-4859-b685-f9ee51f9275e_isrForCrosstalkSources_1031721_9.stderr parsl.dataflow.dflow INFO: Task 1368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa2f5163f3e519c05a6cdaedf8fd7711 with result from task 1368 parsl.dataflow.dflow DEBUG: Task 1368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d2072449-4001-4262-9e4b-2d147ac8fe23_isrForCrosstalkSources_1031725_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d2072449-4001-4262-9e4b-2d147ac8fe23_isrForCrosstalkSources_1031725_3.stderr parsl.dataflow.dflow INFO: Task 1288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a32be9c95f58e549acaa36e90ac56e8 with result from task 1288 parsl.dataflow.dflow DEBUG: Task 1288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0931d5af-cff1-43fa-be80-06988d8d507f_isrForCrosstalkSources_1031737_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0931d5af-cff1-43fa-be80-06988d8d507f_isrForCrosstalkSources_1031737_60.stderr parsl.dataflow.dflow INFO: Task 1370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58a53d544bc10c6a2f4396665c3c4d05 with result from task 1370 parsl.dataflow.dflow DEBUG: Task 1370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/99aae279-adfe-44e8-b02c-c794b8feca0b_isrForCrosstalkSources_1031707_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/99aae279-adfe-44e8-b02c-c794b8feca0b_isrForCrosstalkSources_1031707_50.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be55aa6d2793393f0b7aca0c93b8dd56 with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3b0adab2-c3c0-4310-b5ab-f0b29d35b0e6_isrForCrosstalkSources_1031736_50.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3b0adab2-c3c0-4310-b5ab-f0b29d35b0e6_isrForCrosstalkSources_1031736_50.stderr parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3859079e1b8721c7ae11e96b9ea1f662 with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/9c7e9076-28b1-4e42-960a-c9eb3a483795_isrForCrosstalkSources_1031722_60.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/9c7e9076-28b1-4e42-960a-c9eb3a483795_isrForCrosstalkSources_1031722_60.stderr parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70fdf952183fb3b8fe541890071bf983 with result from task 1072 parsl.dataflow.dflow DEBUG: Task 1072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/817d4905-a3be-4495-ab1d-0749f498e8dd_isrForCrosstalkSources_1031725_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/817d4905-a3be-4495-ab1d-0749f498e8dd_isrForCrosstalkSources_1031725_31.stderr parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed95193e2479890e2d007eec692859bf with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e23eedeb-a499-42a3-bb85-1a683ed314c0_isrForCrosstalkSources_1031717_12.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e23eedeb-a499-42a3-bb85-1a683ed314c0_isrForCrosstalkSources_1031717_12.stderr parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46750b0a24ec7e6b0a4fbe300efbc8c5 with result from task 1073 parsl.dataflow.dflow DEBUG: Task 1073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/3b72fde3-6805-486f-9f02-a9dbb9cd2a28_isrForCrosstalkSources_1031708_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/3b72fde3-6805-486f-9f02-a9dbb9cd2a28_isrForCrosstalkSources_1031708_55.stderr 2024-03-17 04:05:55 proc_lsst.multi:170 [INFO] [multi] job_id 17148084 proc_lsst.multi INFO: [multi] job_id 17148084 2024-03-17 04:05:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 83 2024-03-17 04:05:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148084 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148084 parsl.executors.status_handling DEBUG: Launched block 138 on executor multi with job ID 17148084 parsl.executors.status_handling INFO: Allocated block ID 139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-139 2024-03-17 04:05:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 2024-03-17 04:05:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 83 2024-03-17 04:05:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ef2f849faff86224a3d8d3fe785589f with result from task 970 parsl.dataflow.dflow DEBUG: Task 970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/3030444e-8ce1-4059-bb65-b99ef1844987_isrForCrosstalkSources_1031733_50.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/3030444e-8ce1-4059-bb65-b99ef1844987_isrForCrosstalkSources_1031733_50.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a48a036b189b64da95572fec3eaed7b2 with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ccf95c83-bdb6-452a-b241-d1ad5cf9c095_isrForCrosstalkSources_1031711_9.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ccf95c83-bdb6-452a-b241-d1ad5cf9c095_isrForCrosstalkSources_1031711_9.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d993b97b82ebcd4a659dfe69ca07cae5 with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/16309122-b3e4-4f87-8cff-a980f4dfcde4_isrForCrosstalkSources_1031735_16.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/16309122-b3e4-4f87-8cff-a980f4dfcde4_isrForCrosstalkSources_1031735_16.stderr parsl.dataflow.dflow INFO: Task 1330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95f6de0c3f9b99e4b01a5ccce577f3f3 with result from task 1330 parsl.dataflow.dflow DEBUG: Task 1330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/24c5c019-3fa8-4769-b871-4dd5247155a2_isrForCrosstalkSources_1031716_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/24c5c019-3fa8-4769-b871-4dd5247155a2_isrForCrosstalkSources_1031716_48.stderr parsl.dataflow.dflow INFO: Task 1105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08fbd6b8ecda056b53c74a38fb152847 with result from task 1105 parsl.dataflow.dflow DEBUG: Task 1105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/d23d2b20-c6f5-4d68-9189-e9fdad16e29b_isrForCrosstalkSources_1031732_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/d23d2b20-c6f5-4d68-9189-e9fdad16e29b_isrForCrosstalkSources_1031732_44.stderr parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 091d401f3e9c96b64780a2f23f156ede with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/281db6ac-2ceb-4f74-8caf-2378f2356e4f_isrForCrosstalkSources_1031703_10.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/281db6ac-2ceb-4f74-8caf-2378f2356e4f_isrForCrosstalkSources_1031703_10.stderr parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5adeedd6099ce07c0dd52963d146d564 with result from task 1020 parsl.dataflow.dflow DEBUG: Task 1020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/537d0c6a-d244-4cb1-96e7-17b0aa4a4760_isrForCrosstalkSources_1031710_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/537d0c6a-d244-4cb1-96e7-17b0aa4a4760_isrForCrosstalkSources_1031710_18.stderr parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4764649b2d23c5be4eff75fdb62d583c with result from task 955 parsl.dataflow.dflow DEBUG: Task 955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5366fd81-c367-458d-a8c6-0dea2f97bf88_isrForCrosstalkSources_1031703_54.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5366fd81-c367-458d-a8c6-0dea2f97bf88_isrForCrosstalkSources_1031703_54.stderr parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7592b5a8f5ce2cd15b13addb9d522e06 with result from task 1088 parsl.dataflow.dflow DEBUG: Task 1088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/796e6d26-8357-448e-813a-2fba6a5a6947_isrForCrosstalkSources_1031723_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/796e6d26-8357-448e-813a-2fba6a5a6947_isrForCrosstalkSources_1031723_20.stderr parsl.dataflow.dflow INFO: Task 1349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20ed89d2aabb9d5cda5295aa83345992 with result from task 1349 parsl.dataflow.dflow DEBUG: Task 1349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ce047be-ea32-4d38-a378-7e02d64be75b_isrForCrosstalkSources_1031719_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0ce047be-ea32-4d38-a378-7e02d64be75b_isrForCrosstalkSources_1031719_40.stderr parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6a8a7145363dacb8620d53e8e2c3b6c with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/18c0f02f-8a7c-4763-9310-930153ed2ef4_isrForCrosstalkSources_1031703_38.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/18c0f02f-8a7c-4763-9310-930153ed2ef4_isrForCrosstalkSources_1031703_38.stderr parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02a185937f592db73f24484aeaceb25e with result from task 1098 parsl.dataflow.dflow DEBUG: Task 1098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/210a3a76-f299-4b2e-8d53-b654da98d484_isrForCrosstalkSources_1031700_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/210a3a76-f299-4b2e-8d53-b654da98d484_isrForCrosstalkSources_1031700_23.stderr parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08c1f1d1f6417cc046af17a56d5dfcb3 with result from task 1006 parsl.dataflow.dflow DEBUG: Task 1006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/8c028075-1abb-4f72-a657-a0f0b94c112f_isrForCrosstalkSources_1031710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/8c028075-1abb-4f72-a657-a0f0b94c112f_isrForCrosstalkSources_1031710_31.stderr parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 597d3ade1430261ecfb74b7c93cab6b5 with result from task 1100 parsl.dataflow.dflow DEBUG: Task 1100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f87db46b-039e-4e06-a667-32eed3c5fd91_isrForCrosstalkSources_1031732_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f87db46b-039e-4e06-a667-32eed3c5fd91_isrForCrosstalkSources_1031732_8.stderr parsl.dataflow.dflow INFO: Task 1162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 426fd5631ce2a665b49fd6ffcc5a1232 with result from task 1162 parsl.dataflow.dflow DEBUG: Task 1162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/c869d061-ea41-4089-b978-4fb4802a0a3c_isrForCrosstalkSources_1031737_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/c869d061-ea41-4089-b978-4fb4802a0a3c_isrForCrosstalkSources_1031737_36.stderr parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16ddabc2c1356185e87466408a65e2bd with result from task 1084 parsl.dataflow.dflow DEBUG: Task 1084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/da0fa9aa-f5e5-4c80-ae81-26771e3b4992_isrForCrosstalkSources_1031727_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/da0fa9aa-f5e5-4c80-ae81-26771e3b4992_isrForCrosstalkSources_1031727_27.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 672f5337ed6636ed1ebcfada14a3cfac with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e8fc6a15-a627-4f66-90a0-c343a4e6eecd_isrForCrosstalkSources_1031720_11.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e8fc6a15-a627-4f66-90a0-c343a4e6eecd_isrForCrosstalkSources_1031720_11.stderr parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9107ded8cf2cb24cc6a088ccd4fdcd5a with result from task 1114 parsl.dataflow.dflow DEBUG: Task 1114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bab6070c-be9d-4fab-8f1c-0cb9df3330af_isrForCrosstalkSources_1031734_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bab6070c-be9d-4fab-8f1c-0cb9df3330af_isrForCrosstalkSources_1031734_32.stderr 2024-03-17 04:05:56 proc_lsst.multi:170 [INFO] [multi] job_id 17148085 proc_lsst.multi INFO: [multi] job_id 17148085 2024-03-17 04:05:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 84 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 84 2024-03-17 04:05:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148085 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148085 parsl.executors.status_handling DEBUG: Launched block 139 on executor multi with job ID 17148085 parsl.executors.status_handling INFO: Allocated block ID 140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-140 2024-03-17 04:05:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 2024-03-17 04:05:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 84 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 84 2024-03-17 04:05:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47ba661caad84e76f24607322f1ec0bc with result from task 1120 parsl.dataflow.dflow DEBUG: Task 1120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/d1925aa7-b77e-4ab6-8265-8978f123b868_isrForCrosstalkSources_1031702_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/d1925aa7-b77e-4ab6-8265-8978f123b868_isrForCrosstalkSources_1031702_34.stderr parsl.dataflow.dflow INFO: Task 1146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ed0495d849d959979aad98f97cb26c3 with result from task 1146 parsl.dataflow.dflow DEBUG: Task 1146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3514c62a-05d7-4983-9be3-4890c8db826a_isrForCrosstalkSources_1031716_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/3514c62a-05d7-4983-9be3-4890c8db826a_isrForCrosstalkSources_1031716_26.stderr parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9590846cc9a1469760938faf74ae8a11 with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/de672874-a202-4a79-816d-127650ab3600_isrForCrosstalkSources_1031724_29.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/de672874-a202-4a79-816d-127650ab3600_isrForCrosstalkSources_1031724_29.stderr parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc4880477ad47d323ccb083f432649e0 with result from task 1091 parsl.dataflow.dflow DEBUG: Task 1091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0f1dbb2c-c1b3-4da8-8055-7d12e4b6c09c_isrForCrosstalkSources_1031717_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0f1dbb2c-c1b3-4da8-8055-7d12e4b6c09c_isrForCrosstalkSources_1031717_16.stderr parsl.dataflow.dflow INFO: Task 1351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 363746a740e061a63c45d1212193c8f0 with result from task 1351 parsl.dataflow.dflow DEBUG: Task 1351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/95b619e2-d8cc-4a1b-afb9-bbb17f36c02a_isrForCrosstalkSources_1031723_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/95b619e2-d8cc-4a1b-afb9-bbb17f36c02a_isrForCrosstalkSources_1031723_45.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88aec37ff5e8328756dc5d7b36879b46 with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/12178eb2-e8fe-4672-97a3-b4e909793693_isrForCrosstalkSources_1031720_36.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/12178eb2-e8fe-4672-97a3-b4e909793693_isrForCrosstalkSources_1031720_36.stderr parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88760060029385239f83b5b7f2dba4bb with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/107937c3-eb18-46d7-94a6-28c1f9d3d364_isrForCrosstalkSources_1031715_5.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/107937c3-eb18-46d7-94a6-28c1f9d3d364_isrForCrosstalkSources_1031715_5.stderr parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1649e96f95eefce928c30145fc175711 with result from task 1065 parsl.dataflow.dflow DEBUG: Task 1065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/07f8b70c-4796-4b3f-b930-1b0e57edd38f_isrForCrosstalkSources_1031728_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/07f8b70c-4796-4b3f-b930-1b0e57edd38f_isrForCrosstalkSources_1031728_58.stderr parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 271bfb33f8bbe5913d5bd1682ae692fa with result from task 1102 parsl.dataflow.dflow DEBUG: Task 1102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fb248c28-76b6-4ebe-8eb3-fbbe800877da_isrForCrosstalkSources_1031707_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fb248c28-76b6-4ebe-8eb3-fbbe800877da_isrForCrosstalkSources_1031707_12.stderr parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 976b862b4555f84243d1b15383bc046f with result from task 1052 parsl.dataflow.dflow DEBUG: Task 1052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/285bd959-56e4-4167-9509-76fcc4ed822d_isrForCrosstalkSources_1031723_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/285bd959-56e4-4167-9509-76fcc4ed822d_isrForCrosstalkSources_1031723_29.stderr parsl.dataflow.dflow INFO: Task 1163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f018eb9646df9fb608f3ee092b3ff0a5 with result from task 1163 parsl.dataflow.dflow DEBUG: Task 1163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/97e027d8-0c09-4e46-81a7-f8a9749a5c6a_isrForCrosstalkSources_1031700_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/97e027d8-0c09-4e46-81a7-f8a9749a5c6a_isrForCrosstalkSources_1031700_37.stderr parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5375831be53bc4731896cfad58daff67 with result from task 1016 parsl.dataflow.dflow DEBUG: Task 1016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/93137045-af91-400a-9b38-eaab3e8f2c83_isrForCrosstalkSources_1031702_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/93137045-af91-400a-9b38-eaab3e8f2c83_isrForCrosstalkSources_1031702_35.stderr parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c6dfef37ab3c4607382115cfd75ff8a with result from task 1078 parsl.dataflow.dflow DEBUG: Task 1078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e6bee9d1-efb4-4399-b7d4-464ff7a4d0b9_isrForCrosstalkSources_1031716_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/e6bee9d1-efb4-4399-b7d4-464ff7a4d0b9_isrForCrosstalkSources_1031716_55.stderr parsl.dataflow.dflow INFO: Task 1411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c94f1e588a061cb619bee2d12cd7454 with result from task 1411 parsl.dataflow.dflow DEBUG: Task 1411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ff84eee7-828f-4b76-8b62-4717774915f7_isrForCrosstalkSources_1031708_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ff84eee7-828f-4b76-8b62-4717774915f7_isrForCrosstalkSources_1031708_6.stderr parsl.dataflow.dflow INFO: Task 1262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 174f5c8c39fd556ce702f6a9834dd454 with result from task 1262 parsl.dataflow.dflow DEBUG: Task 1262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5e9e9cde-1764-4ea4-b7ec-7451b26bbad5_isrForCrosstalkSources_1031714_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5e9e9cde-1764-4ea4-b7ec-7451b26bbad5_isrForCrosstalkSources_1031714_12.stderr parsl.dataflow.dflow INFO: Task 1410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 201ff3902fd4c36bfe67cc7eb1554d88 with result from task 1410 parsl.dataflow.dflow DEBUG: Task 1410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4f00fc12-09f4-437f-92a8-5aca19e0d5d8_isrForCrosstalkSources_1031703_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4f00fc12-09f4-437f-92a8-5aca19e0d5d8_isrForCrosstalkSources_1031703_33.stderr 2024-03-17 04:05:57 proc_lsst.multi:170 [INFO] [multi] job_id 17148086 proc_lsst.multi INFO: [multi] job_id 17148086 2024-03-17 04:05:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 85 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 85 2024-03-17 04:05:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148086 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148086 parsl.executors.status_handling DEBUG: Launched block 140 on executor multi with job ID 17148086 parsl.executors.status_handling INFO: Allocated block ID 141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-141 2024-03-17 04:05:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 2024-03-17 04:05:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 85 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 85 2024-03-17 04:05:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11dd60eb894efe40c9b9a3afb5c87688 with result from task 1338 parsl.dataflow.dflow DEBUG: Task 1338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/67aff5ff-3fd9-4926-878b-9ac816a467c1_isrForCrosstalkSources_1031722_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/67aff5ff-3fd9-4926-878b-9ac816a467c1_isrForCrosstalkSources_1031722_30.stderr parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1212a57bfb49345327c60640bca2e33c with result from task 1089 parsl.dataflow.dflow DEBUG: Task 1089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/ab4fa0e9-4040-4dda-ab47-cc05b129fede_isrForCrosstalkSources_1031707_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/ab4fa0e9-4040-4dda-ab47-cc05b129fede_isrForCrosstalkSources_1031707_10.stderr parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb56df261b549c88971e7a56e8694732 with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c4a89fc1-0d79-410e-9125-fb8bc32cc172_isrForCrosstalkSources_1031738_35.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c4a89fc1-0d79-410e-9125-fb8bc32cc172_isrForCrosstalkSources_1031738_35.stderr parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69225de1998ef1e71568282a5795d80b with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e79f8233-62f1-4fb8-98bf-c7c5e6796daa_isrForCrosstalkSources_1031731_61.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e79f8233-62f1-4fb8-98bf-c7c5e6796daa_isrForCrosstalkSources_1031731_61.stderr parsl.dataflow.dflow INFO: Task 1190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcb426937930aa2f906ce747465c4041 with result from task 1190 parsl.dataflow.dflow DEBUG: Task 1190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/ca67070d-6264-4f9b-85cf-54d180e0b221_isrForCrosstalkSources_1031705_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/ca67070d-6264-4f9b-85cf-54d180e0b221_isrForCrosstalkSources_1031705_62.stderr parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf7d9b57f00f6abb07effd174317eb87 with result from task 1115 parsl.dataflow.dflow DEBUG: Task 1115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a4cee620-f244-4344-bc79-1246507fd3ee_isrForCrosstalkSources_1031730_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a4cee620-f244-4344-bc79-1246507fd3ee_isrForCrosstalkSources_1031730_13.stderr parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3361bfbf07e14109c708cb66e662766c with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5743b638-f37f-4cba-8afe-88eab5be4443_isrForCrosstalkSources_1031714_37.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5743b638-f37f-4cba-8afe-88eab5be4443_isrForCrosstalkSources_1031714_37.stderr parsl.dataflow.dflow INFO: Task 1336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeebaf0447124b6c7461de7df0465874 with result from task 1336 parsl.dataflow.dflow DEBUG: Task 1336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5a78d395-2740-41e4-881d-03d9c5b521d3_isrForCrosstalkSources_1031701_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5a78d395-2740-41e4-881d-03d9c5b521d3_isrForCrosstalkSources_1031701_44.stderr parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6d3d62ed993055cef617568839833f4 with result from task 1002 parsl.dataflow.dflow DEBUG: Task 1002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6d69e64d-4227-47f4-a442-300b825f6d09_isrForCrosstalkSources_1031733_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6d69e64d-4227-47f4-a442-300b825f6d09_isrForCrosstalkSources_1031733_61.stderr parsl.dataflow.dflow INFO: Task 1406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81fed2b904d390ad0d396ce270e56bce with result from task 1406 parsl.dataflow.dflow DEBUG: Task 1406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/1c84bf66-64a5-4edd-96df-326388e7daf4_isrForCrosstalkSources_1031723_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/1c84bf66-64a5-4edd-96df-326388e7daf4_isrForCrosstalkSources_1031723_4.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92fdc5b93376b3a38d72cd840b2babc9 with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/70d0c683-0dbc-4ce6-a5b4-a1dedc2ea2ae_isrForCrosstalkSources_1031717_49.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/70d0c683-0dbc-4ce6-a5b4-a1dedc2ea2ae_isrForCrosstalkSources_1031717_49.stderr parsl.dataflow.dflow INFO: Task 1391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a900c9cbad50390e71bb2ea8fd362852 with result from task 1391 parsl.dataflow.dflow DEBUG: Task 1391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2890819b-b35e-4deb-a065-de70fab68261_isrForCrosstalkSources_1031704_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/2890819b-b35e-4deb-a065-de70fab68261_isrForCrosstalkSources_1031704_20.stderr parsl.dataflow.dflow INFO: Task 1151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a8b734e4cfb2cfba43ef8bd3b7ecf73 with result from task 1151 parsl.dataflow.dflow DEBUG: Task 1151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c88dc2bf-eea2-410f-ab34-051c45d265f5_isrForCrosstalkSources_1031716_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c88dc2bf-eea2-410f-ab34-051c45d265f5_isrForCrosstalkSources_1031716_25.stderr parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6e3f574e39aeef58a0e037ef7bfd4f1 with result from task 802 parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/73323920-e1bc-4d3d-9b07-8439cb40dfff_isrForCrosstalkSources_1031708_41.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/73323920-e1bc-4d3d-9b07-8439cb40dfff_isrForCrosstalkSources_1031708_41.stderr parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86efb062986353043f5518eca5dd2304 with result from task 995 parsl.dataflow.dflow DEBUG: Task 995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fc7359d7-2576-4355-a30c-d187172e51c8_isrForCrosstalkSources_1031723_62.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fc7359d7-2576-4355-a30c-d187172e51c8_isrForCrosstalkSources_1031723_62.stderr parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcd92b2624cb9d932507ebd4e436f091 with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/496af609-ad3b-41cb-a214-a1a15d59fba4_isrForCrosstalkSources_1031704_60.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/496af609-ad3b-41cb-a214-a1a15d59fba4_isrForCrosstalkSources_1031704_60.stderr parsl.dataflow.dflow INFO: Task 1424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b48a8b47e21b7bb36a4186b6f07b16bf with result from task 1424 parsl.dataflow.dflow DEBUG: Task 1424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/924e7665-feb2-4a09-87f6-be0dc978c18d_isrForCrosstalkSources_1031700_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/924e7665-feb2-4a09-87f6-be0dc978c18d_isrForCrosstalkSources_1031700_4.stderr parsl.dataflow.dflow INFO: Task 1193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0d7c9694b3a21bc757473e84b91b154 with result from task 1193 parsl.dataflow.dflow DEBUG: Task 1193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6905f6db-1113-4a2d-8422-062ef5ed7db3_isrForCrosstalkSources_1031700_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6905f6db-1113-4a2d-8422-062ef5ed7db3_isrForCrosstalkSources_1031700_5.stderr parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9493fd291b85882badf6ed9db3ab173 with result from task 1085 parsl.dataflow.dflow DEBUG: Task 1085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8fa0d8ea-f63e-453e-b856-c119e17243b4_isrForCrosstalkSources_1031700_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8fa0d8ea-f63e-453e-b856-c119e17243b4_isrForCrosstalkSources_1031700_28.stderr 2024-03-17 04:05:58 proc_lsst.multi:170 [INFO] [multi] job_id 17148087 proc_lsst.multi INFO: [multi] job_id 17148087 2024-03-17 04:05:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 86 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 86 2024-03-17 04:05:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148087 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148087 parsl.executors.status_handling DEBUG: Launched block 141 on executor multi with job ID 17148087 parsl.executors.status_handling INFO: Allocated block ID 142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-142 2024-03-17 04:05:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 2024-03-17 04:05:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 86 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 86 2024-03-17 04:05:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42b15466e693481635f2a75f29d94140 with result from task 1018 parsl.dataflow.dflow DEBUG: Task 1018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/06a36b25-665d-4c31-b61b-3c8359d96b72_isrForCrosstalkSources_1031708_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/06a36b25-665d-4c31-b61b-3c8359d96b72_isrForCrosstalkSources_1031708_49.stderr parsl.dataflow.dflow INFO: Task 1350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 290e0e20aadc83039f5d9e6e8919f1fc with result from task 1350 parsl.dataflow.dflow DEBUG: Task 1350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/00196f13-e10e-43a4-bf3d-e4c0edc113fe_isrForCrosstalkSources_1031709_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/00196f13-e10e-43a4-bf3d-e4c0edc113fe_isrForCrosstalkSources_1031709_9.stderr parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31f622659e99a241554f4333987247cb with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/35f62a9e-3df8-455a-98e0-d4bd39c71f1d_isrForCrosstalkSources_1031699_59.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/35f62a9e-3df8-455a-98e0-d4bd39c71f1d_isrForCrosstalkSources_1031699_59.stderr parsl.dataflow.dflow INFO: Task 1127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6d37489ce887420de79ca681792403b with result from task 1127 parsl.dataflow.dflow DEBUG: Task 1127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bde66494-f855-47b4-bd48-97405980f094_isrForCrosstalkSources_1031734_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bde66494-f855-47b4-bd48-97405980f094_isrForCrosstalkSources_1031734_51.stderr parsl.dataflow.dflow INFO: Task 1197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4ce803f1994227b522e463971e45e56 with result from task 1197 parsl.dataflow.dflow DEBUG: Task 1197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/7847e7cb-4b0a-47ba-9083-41e7b52e514b_isrForCrosstalkSources_1031726_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/7847e7cb-4b0a-47ba-9083-41e7b52e514b_isrForCrosstalkSources_1031726_45.stderr parsl.dataflow.dflow INFO: Task 1421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63f660501c82b6a658af818dd1f229f1 with result from task 1421 parsl.dataflow.dflow DEBUG: Task 1421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d5624d8a-154b-496a-9741-a2c37731a66b_isrForCrosstalkSources_1031726_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d5624d8a-154b-496a-9741-a2c37731a66b_isrForCrosstalkSources_1031726_19.stderr parsl.dataflow.dflow INFO: Task 1118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64f7145985a84d4174b66c169715c489 with result from task 1118 parsl.dataflow.dflow DEBUG: Task 1118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1c062baa-c013-443a-a094-f7937cc14c64_isrForCrosstalkSources_1031701_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1c062baa-c013-443a-a094-f7937cc14c64_isrForCrosstalkSources_1031701_17.stderr parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 498e347c7e689095e1d4f5b549e341b9 with result from task 1009 parsl.dataflow.dflow DEBUG: Task 1009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5e8c5b82-1466-4fd4-91e9-938f4c744e32_isrForCrosstalkSources_1031711_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5e8c5b82-1466-4fd4-91e9-938f4c744e32_isrForCrosstalkSources_1031711_61.stderr parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d83ab934bb79a58b550ed17f45c4508 with result from task 1090 parsl.dataflow.dflow DEBUG: Task 1090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/0622e87d-9068-4954-b99b-e83f41d52337_isrForCrosstalkSources_1031723_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/0622e87d-9068-4954-b99b-e83f41d52337_isrForCrosstalkSources_1031723_37.stderr parsl.dataflow.dflow INFO: Task 1429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8782376623e2d5ffd283b39b223bcb82 with result from task 1429 parsl.dataflow.dflow DEBUG: Task 1429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/29bab00b-7c58-4d0d-b2e6-606267d6099c_isrForCrosstalkSources_1031728_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/29bab00b-7c58-4d0d-b2e6-606267d6099c_isrForCrosstalkSources_1031728_15.stderr parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f4ed311b0278f1b7b0dd6b309a986c7 with result from task 1011 parsl.dataflow.dflow DEBUG: Task 1011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f942457f-55ee-4f73-af7d-8c8478099116_isrForCrosstalkSources_1031709_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f942457f-55ee-4f73-af7d-8c8478099116_isrForCrosstalkSources_1031709_61.stderr parsl.dataflow.dflow INFO: Task 1191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4750333c7ded0ca3be59f96d0106e04d with result from task 1191 parsl.dataflow.dflow DEBUG: Task 1191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d99c8b25-2c4d-4c56-96ec-a430129dec4b_isrForCrosstalkSources_1031722_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d99c8b25-2c4d-4c56-96ec-a430129dec4b_isrForCrosstalkSources_1031722_38.stderr 2024-03-17 04:05:59 proc_lsst.multi:170 [INFO] [multi] job_id 17148088 proc_lsst.multi INFO: [multi] job_id 17148088 2024-03-17 04:05:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 87 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 87 2024-03-17 04:05:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148088 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148088 parsl.executors.status_handling DEBUG: Launched block 142 on executor multi with job ID 17148088 parsl.executors.status_handling INFO: Allocated block ID 143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-143 2024-03-17 04:05:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 2024-03-17 04:05:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:05:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:05:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:05:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:05:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:05:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:05:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:05:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:05:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:05:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:05:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 87 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 87 2024-03-17 04:05:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:05:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea254b5f5105e0ec05bce04808007821 with result from task 1097 parsl.dataflow.dflow DEBUG: Task 1097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/0e068ec8-319c-49b8-9792-844f6a1ca038_isrForCrosstalkSources_1031738_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/0e068ec8-319c-49b8-9792-844f6a1ca038_isrForCrosstalkSources_1031738_52.stderr parsl.dataflow.dflow INFO: Task 1321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6139bb2e98ebb2335be57cb8eb5781c4 with result from task 1321 parsl.dataflow.dflow DEBUG: Task 1321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b4d8deb8-c8cb-45a3-8d29-89635ad20383_isrForCrosstalkSources_1031703_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b4d8deb8-c8cb-45a3-8d29-89635ad20383_isrForCrosstalkSources_1031703_39.stderr parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e13c53a1e2ef465f1c1c9fac72d62bb7 with result from task 1099 parsl.dataflow.dflow DEBUG: Task 1099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/b738912f-9b29-4488-b798-dbeb52c0e98d_isrForCrosstalkSources_1031733_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/b738912f-9b29-4488-b798-dbeb52c0e98d_isrForCrosstalkSources_1031733_30.stderr parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 087bfa58cc789251e48ee0122586923f with result from task 1055 parsl.dataflow.dflow DEBUG: Task 1055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5b8b2abd-ce74-44b8-b791-ebc76ac8a79e_isrForCrosstalkSources_1031735_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5b8b2abd-ce74-44b8-b791-ebc76ac8a79e_isrForCrosstalkSources_1031735_58.stderr parsl.dataflow.dflow INFO: Task 1172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78b58731db75dfe2ebd68df98d500efe with result from task 1172 parsl.dataflow.dflow DEBUG: Task 1172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8f4122dc-b74a-47bd-869a-1d5455ae90cb_isrForCrosstalkSources_1031703_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8f4122dc-b74a-47bd-869a-1d5455ae90cb_isrForCrosstalkSources_1031703_25.stderr parsl.dataflow.dflow INFO: Task 1388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e208558ab0a29466388a3c3a0c90345a with result from task 1388 parsl.dataflow.dflow DEBUG: Task 1388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/def68b47-2985-4c51-be2d-37d829f98eee_isrForCrosstalkSources_1031706_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/def68b47-2985-4c51-be2d-37d829f98eee_isrForCrosstalkSources_1031706_21.stderr parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ae6b6bfb14f5cd7561a1bbafde19ce6 with result from task 1017 parsl.dataflow.dflow DEBUG: Task 1017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/159fccdf-829e-49c8-aaa8-8dc18b055eb5_isrForCrosstalkSources_1031707_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/159fccdf-829e-49c8-aaa8-8dc18b055eb5_isrForCrosstalkSources_1031707_55.stderr parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bfd3e9880dbcb6651907930fbcffc06 with result from task 1019 parsl.dataflow.dflow DEBUG: Task 1019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/baff6d74-0144-4be9-8681-ad75c7532d88_isrForCrosstalkSources_1031724_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/baff6d74-0144-4be9-8681-ad75c7532d88_isrForCrosstalkSources_1031724_55.stderr parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51817f488111e057a812b1ab5b7cd69e with result from task 1087 parsl.dataflow.dflow DEBUG: Task 1087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b97a0cb1-e67f-4691-8e5a-c8ec4584f919_isrForCrosstalkSources_1031713_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b97a0cb1-e67f-4691-8e5a-c8ec4584f919_isrForCrosstalkSources_1031713_30.stderr parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5551e1df58c119627a80f88139bbfd33 with result from task 1103 parsl.dataflow.dflow DEBUG: Task 1103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8f3d0ea5-775a-438a-a609-435951c7b195_isrForCrosstalkSources_1031714_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8f3d0ea5-775a-438a-a609-435951c7b195_isrForCrosstalkSources_1031714_55.stderr parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ab137d9ba07f002dc67f28ab3dcc368 with result from task 1086 parsl.dataflow.dflow DEBUG: Task 1086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/792a809a-f524-4732-b83c-df55aa05541f_isrForCrosstalkSources_1031734_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/792a809a-f524-4732-b83c-df55aa05541f_isrForCrosstalkSources_1031734_60.stderr parsl.dataflow.dflow INFO: Task 1339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87100170e9b6e16bc92d8b49ed3d41f1 with result from task 1339 parsl.dataflow.dflow DEBUG: Task 1339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e610e7e5-a6bb-4c7a-8ef2-875900012d03_isrForCrosstalkSources_1031733_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e610e7e5-a6bb-4c7a-8ef2-875900012d03_isrForCrosstalkSources_1031733_39.stderr parsl.dataflow.dflow INFO: Task 1149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa87eb18a7500909d62081ec5deb5131 with result from task 1149 parsl.dataflow.dflow DEBUG: Task 1149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/de783515-272a-4070-9cd3-182f5e598ae3_isrForCrosstalkSources_1031733_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/de783515-272a-4070-9cd3-182f5e598ae3_isrForCrosstalkSources_1031733_22.stderr parsl.dataflow.dflow INFO: Task 1371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b8eefcf2050f06d8c6a9b7523bf2738 with result from task 1371 parsl.dataflow.dflow DEBUG: Task 1371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/4881429d-040c-4531-8f77-b2a15b9da7d9_isrForCrosstalkSources_1031724_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/4881429d-040c-4531-8f77-b2a15b9da7d9_isrForCrosstalkSources_1031724_13.stderr parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55e3f57d94e490434ef34dbe1198d09c with result from task 1096 parsl.dataflow.dflow DEBUG: Task 1096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e3749196-8131-4148-8c00-bfd9c07b0382_isrForCrosstalkSources_1031717_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e3749196-8131-4148-8c00-bfd9c07b0382_isrForCrosstalkSources_1031717_57.stderr parsl.dataflow.dflow INFO: Task 1161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4193ae0a6579c8757e1157ff9e7f718e with result from task 1161 parsl.dataflow.dflow DEBUG: Task 1161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/cdad2412-a95e-4f21-bdb3-5f7182c65574_isrForCrosstalkSources_1031722_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/cdad2412-a95e-4f21-bdb3-5f7182c65574_isrForCrosstalkSources_1031722_26.stderr parsl.dataflow.dflow INFO: Task 1426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5624eddcb6176f53475095ad1f158a7d with result from task 1426 parsl.dataflow.dflow DEBUG: Task 1426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e03ca61f-2822-4986-83b4-0e793339e5a1_isrForCrosstalkSources_1031730_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e03ca61f-2822-4986-83b4-0e793339e5a1_isrForCrosstalkSources_1031730_52.stderr parsl.dataflow.dflow INFO: Task 1514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cbaa782abc9597a32c6d27d95080cd2 with result from task 1514 parsl.dataflow.dflow DEBUG: Task 1514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/7339aed5-cff6-41b3-a6a8-20eadb824587_isrForCrosstalkSources_1031699_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/7339aed5-cff6-41b3-a6a8-20eadb824587_isrForCrosstalkSources_1031699_27.stderr parsl.dataflow.dflow INFO: Task 1524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81273630ff8ba2b2c7b7d798833cb035 with result from task 1524 parsl.dataflow.dflow DEBUG: Task 1524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1159bbaa-3504-47ba-9ca2-ce8d612b1582_isrForCrosstalkSources_1031735_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/1159bbaa-3504-47ba-9ca2-ce8d612b1582_isrForCrosstalkSources_1031735_22.stderr parsl.dataflow.dflow INFO: Task 1160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78d4b0101d1a3c9ae4bf9395552aa878 with result from task 1160 parsl.dataflow.dflow DEBUG: Task 1160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e565477c-a189-4c71-a7ad-cfe2c76c8aee_isrForCrosstalkSources_1031727_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e565477c-a189-4c71-a7ad-cfe2c76c8aee_isrForCrosstalkSources_1031727_40.stderr parsl.dataflow.dflow INFO: Task 1436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bf3c1eb26d0d2c968a518633f5bdcf4 with result from task 1436 parsl.dataflow.dflow DEBUG: Task 1436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/fa338d43-c26b-489a-8dcd-390d09674d24_isrForCrosstalkSources_1031734_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/fa338d43-c26b-489a-8dcd-390d09674d24_isrForCrosstalkSources_1031734_46.stderr parsl.dataflow.dflow INFO: Task 1441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5969bd27c93f9d160e84459538790ce7 with result from task 1441 parsl.dataflow.dflow DEBUG: Task 1441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/b50949a7-bdd4-49a3-9155-5c80ae45a507_isrForCrosstalkSources_1031727_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/b50949a7-bdd4-49a3-9155-5c80ae45a507_isrForCrosstalkSources_1031727_42.stderr parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cfd9efe689eeb2e8cf694f6cb459223 with result from task 1101 parsl.dataflow.dflow DEBUG: Task 1101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/88af585e-1239-49dc-aa84-1a5c3c6d9efd_isrForCrosstalkSources_1031729_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/88af585e-1239-49dc-aa84-1a5c3c6d9efd_isrForCrosstalkSources_1031729_47.stderr parsl.dataflow.dflow INFO: Task 1195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5796a629984ecaed95d8e47ba2ac647e with result from task 1195 parsl.dataflow.dflow DEBUG: Task 1195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/97e02201-4ad0-487d-a2dc-b83767f40b25_isrForCrosstalkSources_1031711_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/97e02201-4ad0-487d-a2dc-b83767f40b25_isrForCrosstalkSources_1031711_8.stderr parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 904729a2321a79947ec2bcfe0e5238c5 with result from task 1111 parsl.dataflow.dflow DEBUG: Task 1111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/b3385e8a-723d-49f2-b952-daa935765e75_isrForCrosstalkSources_1031719_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/b3385e8a-723d-49f2-b952-daa935765e75_isrForCrosstalkSources_1031719_15.stderr parsl.dataflow.dflow INFO: Task 1131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 077164c47713c9e8276821cd5467d67a with result from task 1131 parsl.dataflow.dflow DEBUG: Task 1131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/507214e5-74fc-40c5-8ab5-4c6e498cc836_isrForCrosstalkSources_1031727_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/507214e5-74fc-40c5-8ab5-4c6e498cc836_isrForCrosstalkSources_1031727_3.stderr parsl.dataflow.dflow INFO: Task 1502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af8566a993e1bb14527f423944dd3bd5 with result from task 1502 parsl.dataflow.dflow DEBUG: Task 1502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a74778ab-a184-4b90-b163-7799a09ab65d_isrForCrosstalkSources_1031711_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a74778ab-a184-4b90-b163-7799a09ab65d_isrForCrosstalkSources_1031711_26.stderr parsl.dataflow.dflow INFO: Task 1442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 872de2820d019bc8c0beebd6ce315d76 with result from task 1442 parsl.dataflow.dflow DEBUG: Task 1442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2ba3cc97-ac4b-4af4-a742-2488ace65aa0_isrForCrosstalkSources_1031734_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2ba3cc97-ac4b-4af4-a742-2488ace65aa0_isrForCrosstalkSources_1031734_16.stderr parsl.dataflow.dflow INFO: Task 1147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca9bf9101e3eab7e32449ba56dea9e8b with result from task 1147 parsl.dataflow.dflow DEBUG: Task 1147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a5ebbb34-445a-4e58-8642-236c94e1dba7_isrForCrosstalkSources_1031713_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a5ebbb34-445a-4e58-8642-236c94e1dba7_isrForCrosstalkSources_1031713_43.stderr parsl.dataflow.dflow INFO: Task 1475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3e0a296a15996ba05a650aa3a92f619 with result from task 1475 parsl.dataflow.dflow DEBUG: Task 1475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fcc1cf16-d754-44ad-9c81-43cefb170d20_isrForCrosstalkSources_1031729_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fcc1cf16-d754-44ad-9c81-43cefb170d20_isrForCrosstalkSources_1031729_32.stderr parsl.dataflow.dflow INFO: Task 1452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf6207aa1738767b598e27099b785b58 with result from task 1452 parsl.dataflow.dflow DEBUG: Task 1452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/b04a5a10-332b-4785-8e23-cfb10f18d61f_isrForCrosstalkSources_1031701_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/b04a5a10-332b-4785-8e23-cfb10f18d61f_isrForCrosstalkSources_1031701_33.stderr 2024-03-17 04:06:01 proc_lsst.multi:170 [INFO] [multi] job_id 17148089 proc_lsst.multi INFO: [multi] job_id 17148089 2024-03-17 04:06:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 88 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 88 2024-03-17 04:06:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148089 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148089 parsl.executors.status_handling DEBUG: Launched block 143 on executor multi with job ID 17148089 parsl.executors.status_handling INFO: Allocated block ID 144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-144 2024-03-17 04:06:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 2024-03-17 04:06:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 88 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 88 2024-03-17 04:06:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3435056f23f4003daaa319db47228af with result from task 1501 parsl.dataflow.dflow DEBUG: Task 1501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e65b9fb3-38f9-4bdf-a12d-8c2f98d4d2af_isrForCrosstalkSources_1031736_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e65b9fb3-38f9-4bdf-a12d-8c2f98d4d2af_isrForCrosstalkSources_1031736_14.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c477464b4a11d65c771766d1a3c33374 with result from task 1401 parsl.dataflow.dflow DEBUG: Task 1401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0aef29e4-4af7-4bf5-833a-55e0721bf234_isrForCrosstalkSources_1031721_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0aef29e4-4af7-4bf5-833a-55e0721bf234_isrForCrosstalkSources_1031721_16.stderr parsl.dataflow.dflow INFO: Task 1407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d8e8fff25da0932d6a384eb8de4bb72 with result from task 1407 parsl.dataflow.dflow DEBUG: Task 1407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4511f23c-8ad7-4670-9fbf-4755a6f3fe52_isrForCrosstalkSources_1031716_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/4511f23c-8ad7-4670-9fbf-4755a6f3fe52_isrForCrosstalkSources_1031716_16.stderr parsl.dataflow.dflow INFO: Task 1119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 504263e12f875570a047e3531db9b9b0 with result from task 1119 parsl.dataflow.dflow DEBUG: Task 1119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/aa50dcbd-b0eb-4d97-be20-b744e5fc415e_isrForCrosstalkSources_1031738_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/aa50dcbd-b0eb-4d97-be20-b744e5fc415e_isrForCrosstalkSources_1031738_10.stderr parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56942ac53e23d148953b145a14ac9f36 with result from task 1113 parsl.dataflow.dflow DEBUG: Task 1113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6da7c709-f61f-4a34-92ae-0deac999dc79_isrForCrosstalkSources_1031721_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6da7c709-f61f-4a34-92ae-0deac999dc79_isrForCrosstalkSources_1031721_10.stderr parsl.dataflow.dflow INFO: Task 1443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 069a99d20184cb8b9fe44ed65de92552 with result from task 1443 parsl.dataflow.dflow DEBUG: Task 1443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/11e1b0b7-9ca5-4998-891d-6ba28d64858b_isrForCrosstalkSources_1031724_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/11e1b0b7-9ca5-4998-891d-6ba28d64858b_isrForCrosstalkSources_1031724_17.stderr parsl.dataflow.dflow INFO: Task 1503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 110fde2b358f7c85dbd27fcfba718f7a with result from task 1503 parsl.dataflow.dflow DEBUG: Task 1503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2e18785a-88b4-4f96-9fe1-c2b3a1ef764f_isrForCrosstalkSources_1031723_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2e18785a-88b4-4f96-9fe1-c2b3a1ef764f_isrForCrosstalkSources_1031723_48.stderr parsl.dataflow.dflow INFO: Task 1512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97dcf13eed71c9754bd26a54b6a410a4 with result from task 1512 parsl.dataflow.dflow DEBUG: Task 1512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/a59d1236-c5e9-4e30-9683-f5fa9a8b018c_isrForCrosstalkSources_1031737_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/a59d1236-c5e9-4e30-9683-f5fa9a8b018c_isrForCrosstalkSources_1031737_56.stderr parsl.dataflow.dflow INFO: Task 1453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8154a2f63345447fa869e9c68790299e with result from task 1453 parsl.dataflow.dflow DEBUG: Task 1453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cef56e85-10aa-44e4-9b99-365063e56b29_isrForCrosstalkSources_1031710_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cef56e85-10aa-44e4-9b99-365063e56b29_isrForCrosstalkSources_1031710_40.stderr parsl.dataflow.dflow INFO: Task 1472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cf56113e2227ff23b18e6e26f019f8c with result from task 1472 parsl.dataflow.dflow DEBUG: Task 1472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/26672e3c-57aa-4819-b1f1-70ebf48bf1ea_isrForCrosstalkSources_1031728_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/26672e3c-57aa-4819-b1f1-70ebf48bf1ea_isrForCrosstalkSources_1031728_19.stderr parsl.dataflow.dflow INFO: Task 1331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2db1f73adcefa3ec6f5969f95793634a with result from task 1331 parsl.dataflow.dflow DEBUG: Task 1331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e0b2580e-aa4f-4f7a-b51d-a217ec1ab4ef_isrForCrosstalkSources_1031725_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e0b2580e-aa4f-4f7a-b51d-a217ec1ab4ef_isrForCrosstalkSources_1031725_9.stderr parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b318856f29674fddbf28fa812b67dc3b with result from task 1112 parsl.dataflow.dflow DEBUG: Task 1112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/99e4f978-ee47-442d-b0c4-2dcb40750ca4_isrForCrosstalkSources_1031730_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/99e4f978-ee47-442d-b0c4-2dcb40750ca4_isrForCrosstalkSources_1031730_11.stderr parsl.dataflow.dflow INFO: Task 1135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b4a7a0e84996e22af263f1fe8185150 with result from task 1135 parsl.dataflow.dflow DEBUG: Task 1135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/604d9c5f-a0d1-4543-abc9-77828dc0c449_isrForCrosstalkSources_1031704_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/604d9c5f-a0d1-4543-abc9-77828dc0c449_isrForCrosstalkSources_1031704_28.stderr parsl.dataflow.dflow INFO: Task 1125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99a360fe7b186328d7c3cd84553f5b80 with result from task 1125 parsl.dataflow.dflow DEBUG: Task 1125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c00fcf87-d245-4132-b739-9d5a0dd6c497_isrForCrosstalkSources_1031702_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/c00fcf87-d245-4132-b739-9d5a0dd6c497_isrForCrosstalkSources_1031702_7.stderr parsl.dataflow.dflow INFO: Task 1445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16fa03ed2e4f12c69d488eab0ed23dd9 with result from task 1445 parsl.dataflow.dflow DEBUG: Task 1445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/9513a4f5-6c05-4c25-b7a4-afe798fd3108_isrForCrosstalkSources_1031709_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/9513a4f5-6c05-4c25-b7a4-afe798fd3108_isrForCrosstalkSources_1031709_16.stderr parsl.dataflow.dflow INFO: Task 1463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec79adaa6896bb416f33fcd6f76793f6 with result from task 1463 parsl.dataflow.dflow DEBUG: Task 1463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e854402f-4b39-4981-bac2-559891dd864f_isrForCrosstalkSources_1031704_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e854402f-4b39-4981-bac2-559891dd864f_isrForCrosstalkSources_1031704_17.stderr parsl.dataflow.dflow INFO: Task 1126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8793531ee81e2288f9a4a1925b77608c with result from task 1126 parsl.dataflow.dflow DEBUG: Task 1126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/813dab32-9ded-4233-baac-18792ac33ca8_isrForCrosstalkSources_1031704_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/813dab32-9ded-4233-baac-18792ac33ca8_isrForCrosstalkSources_1031704_7.stderr parsl.dataflow.dflow INFO: Task 1145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eb04c2ce12f39e9f6f018701bfee348 with result from task 1145 parsl.dataflow.dflow DEBUG: Task 1145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c5fcdb55-8f39-45da-9e48-37c0f8beff8e_isrForCrosstalkSources_1031708_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c5fcdb55-8f39-45da-9e48-37c0f8beff8e_isrForCrosstalkSources_1031708_1.stderr parsl.dataflow.dflow INFO: Task 1200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3861cdc7e5f27049aba21480876eae39 with result from task 1200 parsl.dataflow.dflow DEBUG: Task 1200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/631399f1-ae03-423c-9ae6-db9a2b0cbd43_isrForCrosstalkSources_1031702_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/631399f1-ae03-423c-9ae6-db9a2b0cbd43_isrForCrosstalkSources_1031702_28.stderr parsl.dataflow.dflow INFO: Task 1211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3421603c4d057fba113d4dfdd04a316 with result from task 1211 parsl.dataflow.dflow DEBUG: Task 1211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4cdce307-0563-4156-9d54-4407caf5e4a9_isrForCrosstalkSources_1031717_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4cdce307-0563-4156-9d54-4407caf5e4a9_isrForCrosstalkSources_1031717_24.stderr parsl.dataflow.dflow INFO: Task 1136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a62d7946bf2f2874df2e77cde98123b with result from task 1136 parsl.dataflow.dflow DEBUG: Task 1136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/aeb2a75c-4943-4e5a-be9e-96be6e11bb80_isrForCrosstalkSources_1031723_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/aeb2a75c-4943-4e5a-be9e-96be6e11bb80_isrForCrosstalkSources_1031723_5.stderr parsl.dataflow.dflow INFO: Task 1431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f566d82edb95d8dff2f0856857527595 with result from task 1431 parsl.dataflow.dflow DEBUG: Task 1431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/19e22ecd-c7ae-475d-950b-f997aa5dd725_isrForCrosstalkSources_1031722_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/19e22ecd-c7ae-475d-950b-f997aa5dd725_isrForCrosstalkSources_1031722_4.stderr parsl.dataflow.dflow INFO: Task 1150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fd4c665a7b97d5e553d28c2a093c72d with result from task 1150 parsl.dataflow.dflow DEBUG: Task 1150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1cf320bc-6277-4352-8ac3-8c199495f408_isrForCrosstalkSources_1031701_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1cf320bc-6277-4352-8ac3-8c199495f408_isrForCrosstalkSources_1031701_51.stderr parsl.dataflow.dflow INFO: Task 1572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26ebd2d985d35a369eb3af8667b9d549 with result from task 1572 parsl.dataflow.dflow DEBUG: Task 1572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8c85fab3-4a8c-40da-adf3-db8ee199ad03_isrForCrosstalkSources_1031721_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8c85fab3-4a8c-40da-adf3-db8ee199ad03_isrForCrosstalkSources_1031721_36.stderr parsl.dataflow.dflow INFO: Task 1440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f01cc39ef5d95a83b4ce1ee1959fa36e with result from task 1440 parsl.dataflow.dflow DEBUG: Task 1440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/cdde7a2c-5464-49b7-a71e-955d728ba451_isrForCrosstalkSources_1031705_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/cdde7a2c-5464-49b7-a71e-955d728ba451_isrForCrosstalkSources_1031705_17.stderr parsl.dataflow.dflow INFO: Task 1506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66a947816396f4cfb2601fbc19444a67 with result from task 1506 parsl.dataflow.dflow DEBUG: Task 1506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1cacac24-bc66-456a-ad4e-04b82ab8e7be_isrForCrosstalkSources_1031709_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1cacac24-bc66-456a-ad4e-04b82ab8e7be_isrForCrosstalkSources_1031709_40.stderr parsl.dataflow.dflow INFO: Task 1420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af9f1b501959c9855f214a86ade44a1a with result from task 1420 parsl.dataflow.dflow DEBUG: Task 1420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/49f70376-35e2-40a1-9117-41b47512acd5_isrForCrosstalkSources_1031701_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/49f70376-35e2-40a1-9117-41b47512acd5_isrForCrosstalkSources_1031701_22.stderr parsl.dataflow.dflow INFO: Task 1437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08156a5244dda42801ff3b2995dad075 with result from task 1437 parsl.dataflow.dflow DEBUG: Task 1437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/284d5be8-4144-4b84-bf59-fdc6ed3f1607_isrForCrosstalkSources_1031708_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/284d5be8-4144-4b84-bf59-fdc6ed3f1607_isrForCrosstalkSources_1031708_50.stderr parsl.dataflow.dflow INFO: Task 1439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 005eb81929188f1754f7cd54310ec40f with result from task 1439 parsl.dataflow.dflow DEBUG: Task 1439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/42f2f266-aa06-4f4d-b397-0e289c8e57bf_isrForCrosstalkSources_1031720_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/42f2f266-aa06-4f4d-b397-0e289c8e57bf_isrForCrosstalkSources_1031720_52.stderr parsl.dataflow.dflow INFO: Task 1185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a737e3e523094768636c18e6573ff5c0 with result from task 1185 parsl.dataflow.dflow DEBUG: Task 1185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ebaa52d9-6ddb-4d7b-8846-516835928303_isrForCrosstalkSources_1031708_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ebaa52d9-6ddb-4d7b-8846-516835928303_isrForCrosstalkSources_1031708_39.stderr parsl.dataflow.dflow INFO: Task 1482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69aa12b0609c367deea577537569b8d5 with result from task 1482 parsl.dataflow.dflow DEBUG: Task 1482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/bf152285-688e-42fd-ab4c-e7cc26621abf_isrForCrosstalkSources_1031701_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/bf152285-688e-42fd-ab4c-e7cc26621abf_isrForCrosstalkSources_1031701_15.stderr parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57c5b7bc853e5bf03c1910c9cc71ec10 with result from task 1040 parsl.dataflow.dflow DEBUG: Task 1040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/f4d4ed5c-3640-4c19-867f-a3af103aec50_isrForCrosstalkSources_1031701_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/f4d4ed5c-3640-4c19-867f-a3af103aec50_isrForCrosstalkSources_1031701_49.stderr parsl.dataflow.dflow INFO: Task 1143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61242f41acbaf9dde5cf380bd0a29322 with result from task 1143 parsl.dataflow.dflow DEBUG: Task 1143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/b7358991-784a-4474-bc7c-b79db9c30ac0_isrForCrosstalkSources_1031706_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/b7358991-784a-4474-bc7c-b79db9c30ac0_isrForCrosstalkSources_1031706_43.stderr parsl.dataflow.dflow INFO: Task 1444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 910b79ae3c642b7e6972b100eccbf35e with result from task 1444 parsl.dataflow.dflow DEBUG: Task 1444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c2a3002a-b090-44ad-9136-a61a6ff5ccb8_isrForCrosstalkSources_1031706_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c2a3002a-b090-44ad-9136-a61a6ff5ccb8_isrForCrosstalkSources_1031706_37.stderr parsl.dataflow.dflow INFO: Task 1458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d3a9056e213e7b67419f1b930973f77 with result from task 1458 parsl.dataflow.dflow DEBUG: Task 1458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/e86fb97b-348f-448f-adc2-abbb16164dd8_isrForCrosstalkSources_1031732_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/e86fb97b-348f-448f-adc2-abbb16164dd8_isrForCrosstalkSources_1031732_3.stderr parsl.dataflow.dflow INFO: Task 1438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 782bc49fa885724488d938e62eb98fd5 with result from task 1438 parsl.dataflow.dflow DEBUG: Task 1438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/009458f5-ea18-465a-bf1c-4c1ad5a75f65_isrForCrosstalkSources_1031706_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/009458f5-ea18-465a-bf1c-4c1ad5a75f65_isrForCrosstalkSources_1031706_61.stderr parsl.dataflow.dflow INFO: Task 1183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee3c07cc2d160edfc86f944829a1cda2 with result from task 1183 parsl.dataflow.dflow DEBUG: Task 1183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d9e621e4-1585-4d12-bdf5-ad51a6b16639_isrForCrosstalkSources_1031703_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/d9e621e4-1585-4d12-bdf5-ad51a6b16639_isrForCrosstalkSources_1031703_19.stderr parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea8f61dc79f49e19141682d6ede662d0 with result from task 1032 parsl.dataflow.dflow DEBUG: Task 1032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/159a0800-55b8-4561-a1a4-547b6bc20224_isrForCrosstalkSources_1031721_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/159a0800-55b8-4561-a1a4-547b6bc20224_isrForCrosstalkSources_1031721_52.stderr parsl.dataflow.dflow INFO: Task 1419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38605ca01c5011ee5aed4db0151398af with result from task 1419 parsl.dataflow.dflow DEBUG: Task 1419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c919a639-bd3d-406b-8e1a-2ce5d2bbee66_isrForCrosstalkSources_1031726_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c919a639-bd3d-406b-8e1a-2ce5d2bbee66_isrForCrosstalkSources_1031726_24.stderr parsl.dataflow.dflow INFO: Task 1569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f86991ab80e44355b8943fc2c6698eae with result from task 1569 parsl.dataflow.dflow DEBUG: Task 1569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2f1971d5-fefe-4773-aa69-b936bde7f845_isrForCrosstalkSources_1031734_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2f1971d5-fefe-4773-aa69-b936bde7f845_isrForCrosstalkSources_1031734_22.stderr parsl.dataflow.dflow INFO: Task 1498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0725c3ca580b51068a1fe8c259fbd07 with result from task 1498 parsl.dataflow.dflow DEBUG: Task 1498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/82ab0417-5eab-4f23-8a3f-cd32be7859c9_isrForCrosstalkSources_1031730_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/82ab0417-5eab-4f23-8a3f-cd32be7859c9_isrForCrosstalkSources_1031730_29.stderr parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5016f2c5496430593e9ecc0a96ee5acf with result from task 1035 parsl.dataflow.dflow DEBUG: Task 1035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/26dedbba-e4ca-41b5-a81a-da7393bc1f87_isrForCrosstalkSources_1031721_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/26dedbba-e4ca-41b5-a81a-da7393bc1f87_isrForCrosstalkSources_1031721_47.stderr parsl.dataflow.dflow INFO: Task 1144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d807b351ea74a522ba776338823b421 with result from task 1144 parsl.dataflow.dflow DEBUG: Task 1144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/04ca0c34-8666-4cb9-87ea-c2fb4453a8c6_isrForCrosstalkSources_1031731_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/04ca0c34-8666-4cb9-87ea-c2fb4453a8c6_isrForCrosstalkSources_1031731_7.stderr parsl.dataflow.dflow INFO: Task 1333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa980d427e3e3cba33cd6688e2f5e2ec with result from task 1333 parsl.dataflow.dflow DEBUG: Task 1333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a5e66230-4930-4262-8cdd-3afce305ddc9_isrForCrosstalkSources_1031713_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a5e66230-4930-4262-8cdd-3afce305ddc9_isrForCrosstalkSources_1031713_2.stderr parsl.dataflow.dflow INFO: Task 1129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a0e32fc7103482b6e2d6b3a1696a751 with result from task 1129 parsl.dataflow.dflow DEBUG: Task 1129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/9a21e67d-8967-4ea1-8909-b224f95bd939_isrForCrosstalkSources_1031704_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/9a21e67d-8967-4ea1-8909-b224f95bd939_isrForCrosstalkSources_1031704_57.stderr parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71e1086c116538e80497504324e62541 with result from task 1110 parsl.dataflow.dflow DEBUG: Task 1110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b5056f8f-1747-4fe2-92c6-d126c05feeec_isrForCrosstalkSources_1031714_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b5056f8f-1747-4fe2-92c6-d126c05feeec_isrForCrosstalkSources_1031714_44.stderr parsl.dataflow.dflow INFO: Task 1121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 931bd8acb22109eb98fcc20c236d93d8 with result from task 1121 2024-03-17 04:06:03 proc_lsst.multi:170 [INFO] [multi] job_id 17148090 proc_lsst.multi INFO: [multi] job_id 17148090 2024-03-17 04:06:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 89 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 89 2024-03-17 04:06:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148090 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148090 parsl.dataflow.dflow DEBUG: Task 1121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/0a09788c-7e78-4dba-8e73-e0a8c646a082_isrForCrosstalkSources_1031715_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/0a09788c-7e78-4dba-8e73-e0a8c646a082_isrForCrosstalkSources_1031715_10.stderr parsl.executors.status_handling DEBUG: Launched block 144 on executor multi with job ID 17148090 parsl.executors.status_handling INFO: Allocated block ID 145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-145 2024-03-17 04:06:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 2024-03-17 04:06:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 89 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 89 2024-03-17 04:06:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6e00ee77488356ba92178c686b22756 with result from task 1148 parsl.dataflow.dflow DEBUG: Task 1148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b5161d18-796d-4660-8f0e-f0180c491b69_isrForCrosstalkSources_1031726_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/b5161d18-796d-4660-8f0e-f0180c491b69_isrForCrosstalkSources_1031726_43.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ae0369565a187e3cf32b0ba35a5a524 with result from task 1461 parsl.dataflow.dflow DEBUG: Task 1461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5918cfc7-4dd2-40be-b5db-9660f0ff2c88_isrForCrosstalkSources_1031735_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5918cfc7-4dd2-40be-b5db-9660f0ff2c88_isrForCrosstalkSources_1031735_43.stderr parsl.dataflow.dflow INFO: Task 1176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a265f856d8a9d060fca130eefa9288cf with result from task 1176 parsl.dataflow.dflow DEBUG: Task 1176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/9b3fff15-a0d0-4c4f-84c8-fd4083ebec50_isrForCrosstalkSources_1031738_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/9b3fff15-a0d0-4c4f-84c8-fd4083ebec50_isrForCrosstalkSources_1031738_26.stderr parsl.dataflow.dflow INFO: Task 1199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b9e0a9fc9a05ac25def6a73f442c8c9 with result from task 1199 parsl.dataflow.dflow DEBUG: Task 1199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/67d111ff-f398-4d73-a8bd-4768611b6386_isrForCrosstalkSources_1031725_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/67d111ff-f398-4d73-a8bd-4768611b6386_isrForCrosstalkSources_1031725_42.stderr parsl.dataflow.dflow INFO: Task 1565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fa89e6c7a081ba43894664e6778ce60 with result from task 1565 parsl.dataflow.dflow DEBUG: Task 1565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/377240ee-fdeb-42bd-ac31-36238560ddeb_isrForCrosstalkSources_1031717_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/377240ee-fdeb-42bd-ac31-36238560ddeb_isrForCrosstalkSources_1031717_27.stderr parsl.dataflow.dflow INFO: Task 1130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66934fcfc7a8d1cf9c91097e294fc89b with result from task 1130 parsl.dataflow.dflow DEBUG: Task 1130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e24169f6-5741-4941-b1a5-d1d373b4c42d_isrForCrosstalkSources_1031730_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e24169f6-5741-4941-b1a5-d1d373b4c42d_isrForCrosstalkSources_1031730_58.stderr parsl.dataflow.dflow INFO: Task 1293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 220b7e5a06c1dd662304a19380567a0e with result from task 1293 parsl.dataflow.dflow DEBUG: Task 1293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/9f7980f9-894c-4b72-ac8e-bb4e3fa154af_isrForCrosstalkSources_1031710_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/9f7980f9-894c-4b72-ac8e-bb4e3fa154af_isrForCrosstalkSources_1031710_30.stderr parsl.dataflow.dflow INFO: Task 1589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a0d3a4242588c460daa218925c08993 with result from task 1589 parsl.dataflow.dflow DEBUG: Task 1589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7a4a8190-8281-40e5-9019-50c2bc95ecf5_isrForCrosstalkSources_1031704_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/7a4a8190-8281-40e5-9019-50c2bc95ecf5_isrForCrosstalkSources_1031704_55.stderr parsl.dataflow.dflow INFO: Task 1219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 216b98d281792e85122181e2fb1e7707 with result from task 1219 parsl.dataflow.dflow DEBUG: Task 1219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c656dc57-7c40-4c06-8d97-0c3ffbd9ef30_isrForCrosstalkSources_1031706_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/c656dc57-7c40-4c06-8d97-0c3ffbd9ef30_isrForCrosstalkSources_1031706_22.stderr parsl.dataflow.dflow INFO: Task 1519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51943d4af2e590dd2d8ea1647c3bb0ba with result from task 1519 parsl.dataflow.dflow DEBUG: Task 1519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/939d812c-bf84-4798-8271-92b61512965f_isrForCrosstalkSources_1031719_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/939d812c-bf84-4798-8271-92b61512965f_isrForCrosstalkSources_1031719_22.stderr parsl.dataflow.dflow INFO: Task 1137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46fcd55a4b8b8b275760fdb6c4e2fcfe with result from task 1137 parsl.dataflow.dflow DEBUG: Task 1137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/899b9a50-c6f8-4aa2-a6cb-d41bc59ab94c_isrForCrosstalkSources_1031707_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/899b9a50-c6f8-4aa2-a6cb-d41bc59ab94c_isrForCrosstalkSources_1031707_62.stderr parsl.dataflow.dflow INFO: Task 1596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4db58338be3e7efc0b02684e2f481e3 with result from task 1596 parsl.dataflow.dflow DEBUG: Task 1596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1fa8f7e9-afde-497e-8355-55a74bd0c85c_isrForCrosstalkSources_1031715_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1fa8f7e9-afde-497e-8355-55a74bd0c85c_isrForCrosstalkSources_1031715_44.stderr parsl.dataflow.dflow INFO: Task 1181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c24d876e277d52dbebbf877f346b5a5 with result from task 1181 parsl.dataflow.dflow DEBUG: Task 1181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/54516b10-bb09-43dc-8da4-d68052cc1009_isrForCrosstalkSources_1031728_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/54516b10-bb09-43dc-8da4-d68052cc1009_isrForCrosstalkSources_1031728_46.stderr parsl.dataflow.dflow INFO: Task 1213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e3eb903f4b16fe953decd2d3ac518e1 with result from task 1213 parsl.dataflow.dflow DEBUG: Task 1213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4bbf1243-d1d9-4fc5-a53d-130f751e113b_isrForCrosstalkSources_1031718_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/4bbf1243-d1d9-4fc5-a53d-130f751e113b_isrForCrosstalkSources_1031718_25.stderr parsl.dataflow.dflow INFO: Task 1139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c62031852442535cf492dc312fb722b with result from task 1139 parsl.dataflow.dflow DEBUG: Task 1139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c024fb2a-e6bd-472a-a14a-d8228ac41501_isrForCrosstalkSources_1031726_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/c024fb2a-e6bd-472a-a14a-d8228ac41501_isrForCrosstalkSources_1031726_61.stderr parsl.dataflow.dflow INFO: Task 1236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89ec2ea8f18357481d7a741c87c04bb6 with result from task 1236 parsl.dataflow.dflow DEBUG: Task 1236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ed084f16-2a85-4aeb-8a1c-7f5709a67822_isrForCrosstalkSources_1031713_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ed084f16-2a85-4aeb-8a1c-7f5709a67822_isrForCrosstalkSources_1031713_52.stderr parsl.dataflow.dflow INFO: Task 1202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b8190d99dd559c0eb72c51cf3940fee with result from task 1202 parsl.dataflow.dflow DEBUG: Task 1202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d20dda54-641c-4038-9b46-da37295dcd4b_isrForCrosstalkSources_1031711_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d20dda54-641c-4038-9b46-da37295dcd4b_isrForCrosstalkSources_1031711_59.stderr parsl.dataflow.dflow INFO: Task 1295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8f8e30a88d3e38a34d687037d257a5a with result from task 1295 parsl.dataflow.dflow DEBUG: Task 1295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/b0f61227-9ba6-4304-9887-2480b5aab81f_isrForCrosstalkSources_1031720_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/b0f61227-9ba6-4304-9887-2480b5aab81f_isrForCrosstalkSources_1031720_26.stderr parsl.dataflow.dflow INFO: Task 1123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dff0340b0055ca95ec28d635e535c7f with result from task 1123 parsl.dataflow.dflow DEBUG: Task 1123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/16e89f2d-ceee-4337-baf4-5fa085763497_isrForCrosstalkSources_1031737_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/16e89f2d-ceee-4337-baf4-5fa085763497_isrForCrosstalkSources_1031737_1.stderr parsl.dataflow.dflow INFO: Task 1574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dd4d59a9978d919c625d11d5fb34e18 with result from task 1574 parsl.dataflow.dflow DEBUG: Task 1574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/e8948ac1-4fbe-4e55-aa3d-3d52d376a11d_isrForCrosstalkSources_1031713_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/e8948ac1-4fbe-4e55-aa3d-3d52d376a11d_isrForCrosstalkSources_1031713_57.stderr 2024-03-17 04:06:04 proc_lsst.multi:170 [INFO] [multi] job_id 17148091 proc_lsst.multi INFO: [multi] job_id 17148091 2024-03-17 04:06:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 90 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 90 2024-03-17 04:06:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148091 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148091 parsl.executors.status_handling DEBUG: Launched block 145 on executor multi with job ID 17148091 parsl.executors.status_handling INFO: Allocated block ID 146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-146 2024-03-17 04:06:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 2024-03-17 04:06:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 90 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 90 2024-03-17 04:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7edeacc6e903b4c0553b28e8661d81b8 with result from task 1292 parsl.dataflow.dflow DEBUG: Task 1292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d83f2e92-1e57-4607-b673-b3b6e6565ff5_isrForCrosstalkSources_1031711_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d83f2e92-1e57-4607-b673-b3b6e6565ff5_isrForCrosstalkSources_1031711_14.stderr parsl.dataflow.dflow INFO: Task 1597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6426de473558f6c40f975a5a2d3a7eb7 with result from task 1597 parsl.dataflow.dflow DEBUG: Task 1597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d7eb2f56-e061-4f56-8278-31cb3ead396f_isrForCrosstalkSources_1031722_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d7eb2f56-e061-4f56-8278-31cb3ead396f_isrForCrosstalkSources_1031722_44.stderr parsl.dataflow.dflow INFO: Task 1348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d119017096fa7e8f2f39fc22a181113 with result from task 1348 parsl.dataflow.dflow DEBUG: Task 1348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/976a0810-31ac-4cac-be55-1501a7f7ab91_isrForCrosstalkSources_1031716_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/976a0810-31ac-4cac-be55-1501a7f7ab91_isrForCrosstalkSources_1031716_7.stderr parsl.dataflow.dflow INFO: Task 1214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76b2a2b764f016bec8a6b8887367ffa0 with result from task 1214 parsl.dataflow.dflow DEBUG: Task 1214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/19357293-f5af-4b98-a551-68f1f4fcd2b3_isrForCrosstalkSources_1031710_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/19357293-f5af-4b98-a551-68f1f4fcd2b3_isrForCrosstalkSources_1031710_21.stderr parsl.dataflow.dflow INFO: Task 1586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dd754c1349a3e34d059a91748a13395 with result from task 1586 parsl.dataflow.dflow DEBUG: Task 1586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b7822bca-9659-44b7-8e60-14df1547ee4c_isrForCrosstalkSources_1031707_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b7822bca-9659-44b7-8e60-14df1547ee4c_isrForCrosstalkSources_1031707_20.stderr parsl.dataflow.dflow INFO: Task 1483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4d9384bb2f65f067ecbd97cc7640212 with result from task 1483 parsl.dataflow.dflow DEBUG: Task 1483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3221af40-4101-44fb-8b08-0bd594cc8af5_isrForCrosstalkSources_1031731_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3221af40-4101-44fb-8b08-0bd594cc8af5_isrForCrosstalkSources_1031731_17.stderr parsl.dataflow.dflow INFO: Task 1133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9866028d501a4ea0812083eb87814079 with result from task 1133 parsl.dataflow.dflow DEBUG: Task 1133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7b6ff05e-f455-449c-853e-ad5807a8a977_isrForCrosstalkSources_1031729_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7b6ff05e-f455-449c-853e-ad5807a8a977_isrForCrosstalkSources_1031729_53.stderr parsl.dataflow.dflow INFO: Task 1595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a372cde2a77e4d12c395d5a116a29327 with result from task 1595 parsl.dataflow.dflow DEBUG: Task 1595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/ab1d8340-b193-4358-b10f-c61865941964_isrForCrosstalkSources_1031719_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/ab1d8340-b193-4358-b10f-c61865941964_isrForCrosstalkSources_1031719_14.stderr parsl.dataflow.dflow INFO: Task 1485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cbcb4478106b6e9bcb388e7cd943c18 with result from task 1485 parsl.dataflow.dflow DEBUG: Task 1485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/f4f6d9fd-4576-407a-a081-5eac434f0ed3_isrForCrosstalkSources_1031722_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/f4f6d9fd-4576-407a-a081-5eac434f0ed3_isrForCrosstalkSources_1031722_46.stderr parsl.dataflow.dflow INFO: Task 1296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33dca735c29bdc31bdcad223ecafab49 with result from task 1296 parsl.dataflow.dflow DEBUG: Task 1296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/e03b3abf-3b56-489d-86b6-5fd7bf466b4d_isrForCrosstalkSources_1031713_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/e03b3abf-3b56-489d-86b6-5fd7bf466b4d_isrForCrosstalkSources_1031713_4.stderr parsl.dataflow.dflow INFO: Task 1417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8577099cf9c12a215cb3de794c26372 with result from task 1417 parsl.dataflow.dflow DEBUG: Task 1417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/3da43e6d-2951-4852-b43b-bd6331d18f4d_isrForCrosstalkSources_1031714_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/3da43e6d-2951-4852-b43b-bd6331d18f4d_isrForCrosstalkSources_1031714_45.stderr parsl.dataflow.dflow INFO: Task 1581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 811afbc7afe4e2c115de096946cf3822 with result from task 1581 parsl.dataflow.dflow DEBUG: Task 1581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/aaebfb9f-393e-449d-a399-767b60daef04_isrForCrosstalkSources_1031714_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/aaebfb9f-393e-449d-a399-767b60daef04_isrForCrosstalkSources_1031714_20.stderr parsl.dataflow.dflow INFO: Task 1593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7b2cf51a2863e4d6414fc3bf3f03f8c with result from task 1593 parsl.dataflow.dflow DEBUG: Task 1593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/63d9d0d7-e7b1-4024-818c-b5cde3bdd317_isrForCrosstalkSources_1031703_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/63d9d0d7-e7b1-4024-818c-b5cde3bdd317_isrForCrosstalkSources_1031703_9.stderr parsl.dataflow.dflow INFO: Task 1534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11cd73ad363fba31cb2b21dec3cff689 with result from task 1534 parsl.dataflow.dflow DEBUG: Task 1534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/6c0aa16d-0f95-4faa-8a6c-2cc84264f616_isrForCrosstalkSources_1031725_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/6c0aa16d-0f95-4faa-8a6c-2cc84264f616_isrForCrosstalkSources_1031725_26.stderr parsl.dataflow.dflow INFO: Task 1528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56e09a399256cbddbdf6f188d1f5c018 with result from task 1528 parsl.dataflow.dflow DEBUG: Task 1528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5d551355-0dbf-4fb0-947e-5b23a6e1d5da_isrForCrosstalkSources_1031727_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5d551355-0dbf-4fb0-947e-5b23a6e1d5da_isrForCrosstalkSources_1031727_25.stderr parsl.dataflow.dflow INFO: Task 1606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b90e39aa85dc19db74a70eeb81b8ad58 with result from task 1606 parsl.dataflow.dflow DEBUG: Task 1606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/95e3352a-8c19-4ede-b688-13d844aafd55_isrForCrosstalkSources_1031736_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/95e3352a-8c19-4ede-b688-13d844aafd55_isrForCrosstalkSources_1031736_23.stderr parsl.dataflow.dflow INFO: Task 1471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f6d8c26eab4e3067a5774aa9cf94ca3 with result from task 1471 parsl.dataflow.dflow DEBUG: Task 1471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/af5c3141-4522-4783-bbc8-f70104e05676_isrForCrosstalkSources_1031711_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/af5c3141-4522-4783-bbc8-f70104e05676_isrForCrosstalkSources_1031711_22.stderr parsl.dataflow.dflow INFO: Task 1173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3acafdce96c9d496dfcdb9f8aaa3612c with result from task 1173 parsl.dataflow.dflow DEBUG: Task 1173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1631fbb5-4ffe-4087-a212-427654300aa2_isrForCrosstalkSources_1031711_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1631fbb5-4ffe-4087-a212-427654300aa2_isrForCrosstalkSources_1031711_28.stderr parsl.dataflow.dflow INFO: Task 1578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee54ab59deb5ed4bf18be5050eadfe83 with result from task 1578 parsl.dataflow.dflow DEBUG: Task 1578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/d529e2fb-a748-462c-8bc1-054972944274_isrForCrosstalkSources_1031713_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/d529e2fb-a748-462c-8bc1-054972944274_isrForCrosstalkSources_1031713_46.stderr parsl.dataflow.dflow INFO: Task 1486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb60b4e74c8a4ac51298fdc4a5954863 with result from task 1486 parsl.dataflow.dflow DEBUG: Task 1486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/7f83e4c9-a3bf-48ec-8de5-07b19541eb61_isrForCrosstalkSources_1031736_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/7f83e4c9-a3bf-48ec-8de5-07b19541eb61_isrForCrosstalkSources_1031736_46.stderr parsl.dataflow.dflow INFO: Task 1600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67f5358be6dac4c264eaab964392c796 with result from task 1600 parsl.dataflow.dflow DEBUG: Task 1600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/355f11da-ebc5-4eb0-a638-da0c2a20dece_isrForCrosstalkSources_1031725_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/355f11da-ebc5-4eb0-a638-da0c2a20dece_isrForCrosstalkSources_1031725_24.stderr 2024-03-17 04:06:05 proc_lsst.multi:170 [INFO] [multi] job_id 17148092 proc_lsst.multi INFO: [multi] job_id 17148092 2024-03-17 04:06:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 91 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 91 2024-03-17 04:06:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148092 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148092 parsl.executors.status_handling DEBUG: Launched block 146 on executor multi with job ID 17148092 parsl.executors.status_handling INFO: Allocated block ID 147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-147 2024-03-17 04:06:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 2024-03-17 04:06:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 91 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 91 2024-03-17 04:06:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c77b3dab1cb09fd291c6753bb0f11f8 with result from task 1042 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 1042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4a5a7275-514c-4a80-90f4-5c9b19f4594b_isrForCrosstalkSources_1031719_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4a5a7275-514c-4a80-90f4-5c9b19f4594b_isrForCrosstalkSources_1031719_61.stderr parsl.dataflow.dflow INFO: Task 1430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a08a2c82a24605e1ce714d1934ede1c with result from task 1430 parsl.dataflow.dflow DEBUG: Task 1430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8147d972-eaaf-48ac-b052-f383dec9ac92_isrForCrosstalkSources_1031714_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8147d972-eaaf-48ac-b052-f383dec9ac92_isrForCrosstalkSources_1031714_6.stderr parsl.dataflow.dflow INFO: Task 1543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec2712d71fd4e1ceff9af65f251cbe4a with result from task 1543 parsl.dataflow.dflow DEBUG: Task 1543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/a30fca6c-91e7-475c-8886-8cf460d7b888_isrForCrosstalkSources_1031734_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/a30fca6c-91e7-475c-8886-8cf460d7b888_isrForCrosstalkSources_1031734_25.stderr parsl.dataflow.dflow INFO: Task 1467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 800184e79683e9c2d2e19798415aad44 with result from task 1467 parsl.dataflow.dflow DEBUG: Task 1467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8c650af3-2bdc-4640-bafa-f7f8b3dce07a_isrForCrosstalkSources_1031714_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/8c650af3-2bdc-4640-bafa-f7f8b3dce07a_isrForCrosstalkSources_1031714_15.stderr parsl.dataflow.dflow INFO: Task 1423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8d1420de3f6217d1147a2316b6a60cd with result from task 1423 parsl.dataflow.dflow DEBUG: Task 1423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/78cfe645-cc96-420c-a8a2-c9de30dd0177_isrForCrosstalkSources_1031714_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/78cfe645-cc96-420c-a8a2-c9de30dd0177_isrForCrosstalkSources_1031714_56.stderr parsl.dataflow.dflow INFO: Task 1579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fcaff7d298fcc5c9cddf2deb92fe597 with result from task 1579 parsl.dataflow.dflow DEBUG: Task 1579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fdfd91b5-2c8d-4c09-966d-0b0b3cd51383_isrForCrosstalkSources_1031723_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/fdfd91b5-2c8d-4c09-966d-0b0b3cd51383_isrForCrosstalkSources_1031723_16.stderr parsl.dataflow.dflow INFO: Task 1491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13e5025a6a16620819cb60ef8cf1402a with result from task 1491 parsl.dataflow.dflow DEBUG: Task 1491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/4e54b04c-a956-4678-a436-48627b58bf23_isrForCrosstalkSources_1031736_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/4e54b04c-a956-4678-a436-48627b58bf23_isrForCrosstalkSources_1031736_19.stderr parsl.dataflow.dflow INFO: Task 1516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e82a74f913d16c7ce9b5b63184157e8 with result from task 1516 parsl.dataflow.dflow DEBUG: Task 1516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/135a0652-f61f-4289-aa2f-fb2d8ed6def2_isrForCrosstalkSources_1031708_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/135a0652-f61f-4289-aa2f-fb2d8ed6def2_isrForCrosstalkSources_1031708_56.stderr parsl.dataflow.dflow INFO: Task 1402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5030d14cb27476e142f93f0334c0a55b with result from task 1402 parsl.dataflow.dflow DEBUG: Task 1402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/69b3aa9f-1426-4168-b703-6e3db627a682_isrForCrosstalkSources_1031704_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/69b3aa9f-1426-4168-b703-6e3db627a682_isrForCrosstalkSources_1031704_32.stderr 2024-03-17 04:06:06 proc_lsst.multi:170 [INFO] [multi] job_id 17148093 proc_lsst.multi INFO: [multi] job_id 17148093 2024-03-17 04:06:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 92 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 92 2024-03-17 04:06:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148093 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148093 parsl.executors.status_handling DEBUG: Launched block 147 on executor multi with job ID 17148093 parsl.executors.status_handling INFO: Allocated block ID 148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-148 2024-03-17 04:06:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 2024-03-17 04:06:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 92 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 92 2024-03-17 04:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f119477645c7350797d44bc7bce135ae with result from task 1588 parsl.dataflow.dflow DEBUG: Task 1588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/85b6bd53-aac1-4fe3-91e1-360a538774fe_isrForCrosstalkSources_1031703_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/85b6bd53-aac1-4fe3-91e1-360a538774fe_isrForCrosstalkSources_1031703_51.stderr parsl.dataflow.dflow INFO: Task 1435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95aa581bbed0876b3089970417a76583 with result from task 1435 parsl.dataflow.dflow DEBUG: Task 1435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4daffc18-34c6-403a-ac91-69dbe1de904e_isrForCrosstalkSources_1031699_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4daffc18-34c6-403a-ac91-69dbe1de904e_isrForCrosstalkSources_1031699_20.stderr parsl.dataflow.dflow INFO: Task 1557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41cfbb35237762ae0a0bdd9b96fec466 with result from task 1557 parsl.dataflow.dflow DEBUG: Task 1557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a06e77c2-5c03-4f5e-90c3-9d878c0078ae_isrForCrosstalkSources_1031721_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/a06e77c2-5c03-4f5e-90c3-9d878c0078ae_isrForCrosstalkSources_1031721_33.stderr parsl.dataflow.dflow INFO: Task 1446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ee1c7eac0842e7fabe9d50285dcfee1 with result from task 1446 parsl.dataflow.dflow DEBUG: Task 1446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/57b5bca3-014c-4cc7-a453-061a65f46c09_isrForCrosstalkSources_1031717_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/57b5bca3-014c-4cc7-a453-061a65f46c09_isrForCrosstalkSources_1031717_28.stderr parsl.dataflow.dflow INFO: Task 1287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 343684330d79a073a6c68f7a7cc3fd05 with result from task 1287 parsl.dataflow.dflow DEBUG: Task 1287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/ce7256f0-2463-42d6-aaf1-b9b189af396b_isrForCrosstalkSources_1031703_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/ce7256f0-2463-42d6-aaf1-b9b189af396b_isrForCrosstalkSources_1031703_31.stderr parsl.dataflow.dflow INFO: Task 1294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e3820610fc889001309b2a742b4e392 with result from task 1294 parsl.dataflow.dflow DEBUG: Task 1294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/6e79e23c-454f-4b70-af05-b954b3fb34cb_isrForCrosstalkSources_1031717_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/6e79e23c-454f-4b70-af05-b954b3fb34cb_isrForCrosstalkSources_1031717_46.stderr parsl.dataflow.dflow INFO: Task 1508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e17780d5e6c793f547cfd7cb12d8309 with result from task 1508 parsl.dataflow.dflow DEBUG: Task 1508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d78d4273-d8a1-4b37-a7c5-550b9f2ef92e_isrForCrosstalkSources_1031726_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d78d4273-d8a1-4b37-a7c5-550b9f2ef92e_isrForCrosstalkSources_1031726_47.stderr parsl.dataflow.dflow INFO: Task 1511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90c86cd838a165a06470ec7af239f4f4 with result from task 1511 parsl.dataflow.dflow DEBUG: Task 1511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3a4844e5-9f66-4671-8292-b40eab3fbf9e_isrForCrosstalkSources_1031731_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/3a4844e5-9f66-4671-8292-b40eab3fbf9e_isrForCrosstalkSources_1031731_30.stderr parsl.dataflow.dflow INFO: Task 1218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1278c4a73b58efcae2a898386455adf9 with result from task 1218 parsl.dataflow.dflow DEBUG: Task 1218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f9757a61-a497-4bfd-915d-3b9ba9932f64_isrForCrosstalkSources_1031710_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f9757a61-a497-4bfd-915d-3b9ba9932f64_isrForCrosstalkSources_1031710_62.stderr parsl.dataflow.dflow INFO: Task 1245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f31bf67ca1d7289cb41822c3e7b891e9 with result from task 1245 parsl.dataflow.dflow DEBUG: Task 1245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/da2a8fa6-946e-48e0-8809-4576c1df689d_isrForCrosstalkSources_1031738_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/da2a8fa6-946e-48e0-8809-4576c1df689d_isrForCrosstalkSources_1031738_38.stderr parsl.dataflow.dflow INFO: Task 1403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67b23e86885aa1427cee1d7609a03cfa with result from task 1403 parsl.dataflow.dflow DEBUG: Task 1403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/0a53845c-96bf-440f-87f6-1d9910605166_isrForCrosstalkSources_1031723_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/0a53845c-96bf-440f-87f6-1d9910605166_isrForCrosstalkSources_1031723_47.stderr 2024-03-17 04:06:07 proc_lsst.multi:170 [INFO] [multi] job_id 17148094 proc_lsst.multi INFO: [multi] job_id 17148094 2024-03-17 04:06:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 93 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 93 2024-03-17 04:06:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148094 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148094 parsl.executors.status_handling DEBUG: Launched block 148 on executor multi with job ID 17148094 parsl.executors.status_handling INFO: Allocated block ID 149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-149 2024-03-17 04:06:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 2024-03-17 04:06:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 93 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 93 2024-03-17 04:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 396cfc2150c7f50ccd823fbc77ecf828 with result from task 1215 parsl.dataflow.dflow DEBUG: Task 1215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/7e9d0ec8-087a-4528-bd58-28159e07461b_isrForCrosstalkSources_1031709_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/7e9d0ec8-087a-4528-bd58-28159e07461b_isrForCrosstalkSources_1031709_10.stderr parsl.dataflow.dflow INFO: Task 1290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc5c43c26cf3f905f6faf4b954489e30 with result from task 1290 parsl.dataflow.dflow DEBUG: Task 1290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/353cb1fc-50b7-4015-88bf-ff9feb222fa9_isrForCrosstalkSources_1031704_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/353cb1fc-50b7-4015-88bf-ff9feb222fa9_isrForCrosstalkSources_1031704_11.stderr parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b92b6a32b8298f55ca880ceba0bc684c with result from task 1038 parsl.dataflow.dflow DEBUG: Task 1038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1763f9e8-a377-40d2-a59a-2318b437a6b3_isrForCrosstalkSources_1031706_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/1763f9e8-a377-40d2-a59a-2318b437a6b3_isrForCrosstalkSources_1031706_59.stderr parsl.dataflow.dflow INFO: Task 1468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d48e6bc41f11b14e2d4711629f1dc59 with result from task 1468 parsl.dataflow.dflow DEBUG: Task 1468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/309fad20-7589-4033-b40f-07d9c31f5103_isrForCrosstalkSources_1031710_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/309fad20-7589-4033-b40f-07d9c31f5103_isrForCrosstalkSources_1031710_56.stderr parsl.dataflow.dflow INFO: Task 1476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e7e876a730d49ee3c4bfd4e1e88caa3 with result from task 1476 parsl.dataflow.dflow DEBUG: Task 1476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/f3ea38f0-d265-4644-8f06-aa740b49806e_isrForCrosstalkSources_1031704_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/f3ea38f0-d265-4644-8f06-aa740b49806e_isrForCrosstalkSources_1031704_58.stderr parsl.dataflow.dflow INFO: Task 1188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93b9ed1bcb242fc61f1ffe3b3b59f3ac with result from task 1188 parsl.dataflow.dflow DEBUG: Task 1188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f92f5092-9e06-4560-a4ae-93c1212ec3e8_isrForCrosstalkSources_1031705_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f92f5092-9e06-4560-a4ae-93c1212ec3e8_isrForCrosstalkSources_1031705_38.stderr 2024-03-17 04:06:07 proc_lsst.multi:170 [INFO] [multi] job_id 17148095 proc_lsst.multi INFO: [multi] job_id 17148095 2024-03-17 04:06:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 94 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 94 2024-03-17 04:06:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148095 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148095 parsl.executors.status_handling DEBUG: Launched block 149 on executor multi with job ID 17148095 parsl.executors.status_handling INFO: Allocated block ID 150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-150 2024-03-17 04:06:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 2024-03-17 04:06:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 94 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 94 2024-03-17 04:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7a626bed4500af410e6f9acb862f014 with result from task 1536 parsl.dataflow.dflow DEBUG: Task 1536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/30b2c9c2-1a09-4644-bcb8-5c523a49c156_isrForCrosstalkSources_1031716_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/30b2c9c2-1a09-4644-bcb8-5c523a49c156_isrForCrosstalkSources_1031716_20.stderr parsl.dataflow.dflow INFO: Task 1500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38c6bb89b7a2ebb885d9fc9847f31291 with result from task 1500 parsl.dataflow.dflow DEBUG: Task 1500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e923b42a-66db-4356-986f-41acd727d818_isrForCrosstalkSources_1031701_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e923b42a-66db-4356-986f-41acd727d818_isrForCrosstalkSources_1031701_4.stderr parsl.dataflow.dflow INFO: Task 1192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0121b6be891164e66a8f3fc1d908e00 with result from task 1192 parsl.dataflow.dflow DEBUG: Task 1192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ec2a45be-e1ae-47a7-998a-fd5aa71a91c1_isrForCrosstalkSources_1031708_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ec2a45be-e1ae-47a7-998a-fd5aa71a91c1_isrForCrosstalkSources_1031708_59.stderr parsl.dataflow.dflow INFO: Task 1559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eac932a473509d62aaf8b8a29886b386 with result from task 1559 parsl.dataflow.dflow DEBUG: Task 1559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/257022d2-4d58-4b6a-a8d0-2b18db6f1870_isrForCrosstalkSources_1031709_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/257022d2-4d58-4b6a-a8d0-2b18db6f1870_isrForCrosstalkSources_1031709_7.stderr parsl.dataflow.dflow INFO: Task 1489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f00f89d3bcad24bcf4420c00e4a7f38 with result from task 1489 parsl.dataflow.dflow DEBUG: Task 1489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ac628fc8-ff61-428f-b10b-6d3d77173e4c_isrForCrosstalkSources_1031724_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ac628fc8-ff61-428f-b10b-6d3d77173e4c_isrForCrosstalkSources_1031724_39.stderr parsl.dataflow.dflow INFO: Task 1523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f084ecf8ee61eff4afc0f88c349255d4 with result from task 1523 parsl.dataflow.dflow DEBUG: Task 1523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/adbe1f5d-c051-4762-ad8f-7d03c24e1819_isrForCrosstalkSources_1031720_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/adbe1f5d-c051-4762-ad8f-7d03c24e1819_isrForCrosstalkSources_1031720_14.stderr parsl.dataflow.dflow INFO: Task 1408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1181496afccd59e46a527bca53dff823 with result from task 1408 parsl.dataflow.dflow DEBUG: Task 1408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ddcc9d8e-58a6-4e2b-ade6-87060a12e055_isrForCrosstalkSources_1031715_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ddcc9d8e-58a6-4e2b-ade6-87060a12e055_isrForCrosstalkSources_1031715_17.stderr parsl.dataflow.dflow INFO: Task 1174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f15cdde5681194d8cbf62a5a1306475f with result from task 1174 parsl.dataflow.dflow DEBUG: Task 1174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/a90b419a-df79-4645-8ae6-889b878f9c98_isrForCrosstalkSources_1031705_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/a90b419a-df79-4645-8ae6-889b878f9c98_isrForCrosstalkSources_1031705_42.stderr parsl.dataflow.dflow INFO: Task 1474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3a6a3e0fb0275cc4602217152f7ca8e with result from task 1474 parsl.dataflow.dflow DEBUG: Task 1474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/11b410f1-9df9-4a6a-a4ee-9b605168bf73_isrForCrosstalkSources_1031724_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/11b410f1-9df9-4a6a-a4ee-9b605168bf73_isrForCrosstalkSources_1031724_18.stderr parsl.dataflow.dflow INFO: Task 1389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10735740433b2a821ac20f5d73a07d25 with result from task 1389 parsl.dataflow.dflow DEBUG: Task 1389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/06f47bbf-fe9a-40d9-9cf3-c3660e8dcaea_isrForCrosstalkSources_1031721_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/06f47bbf-fe9a-40d9-9cf3-c3660e8dcaea_isrForCrosstalkSources_1031721_42.stderr parsl.dataflow.dflow INFO: Task 1196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24fd469456e24e2027b26b0c6fdec66e with result from task 1196 parsl.dataflow.dflow DEBUG: Task 1196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/903e1325-4ebf-43a0-aa66-14be3034d212_isrForCrosstalkSources_1031710_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/903e1325-4ebf-43a0-aa66-14be3034d212_isrForCrosstalkSources_1031710_50.stderr parsl.dataflow.dflow INFO: Task 1591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 641e706488211fbe4d5580a032b84baf with result from task 1591 parsl.dataflow.dflow DEBUG: Task 1591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3110feba-8386-4901-85fc-be244574ec2d_isrForCrosstalkSources_1031713_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3110feba-8386-4901-85fc-be244574ec2d_isrForCrosstalkSources_1031713_13.stderr 2024-03-17 04:06:08 proc_lsst.multi:170 [INFO] [multi] job_id 17148096 proc_lsst.multi INFO: [multi] job_id 17148096 2024-03-17 04:06:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 95 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 95 2024-03-17 04:06:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148096 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148096 parsl.executors.status_handling DEBUG: Launched block 150 on executor multi with job ID 17148096 parsl.executors.status_handling INFO: Allocated block ID 151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-151 2024-03-17 04:06:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 2024-03-17 04:06:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 95 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 95 2024-03-17 04:06:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1ff150b559ac779ba467c57f97eeb7a with result from task 1409 parsl.dataflow.dflow DEBUG: Task 1409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/14a5a416-df48-483e-bc12-115335f56fed_isrForCrosstalkSources_1031699_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/14a5a416-df48-483e-bc12-115335f56fed_isrForCrosstalkSources_1031699_2.stderr parsl.dataflow.dflow INFO: Task 1216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd583e3c855759b7c52cad6ebac881e1 with result from task 1216 parsl.dataflow.dflow DEBUG: Task 1216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5ff62239-594c-4c03-a3d6-446ef4e82f50_isrForCrosstalkSources_1031705_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5ff62239-594c-4c03-a3d6-446ef4e82f50_isrForCrosstalkSources_1031705_41.stderr parsl.dataflow.dflow INFO: Task 1628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdc52990f0b667bd37b2cc6d14f110f1 with result from task 1628 parsl.dataflow.dflow DEBUG: Task 1628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0c106321-bdc2-476f-b29e-a77122025938_isrForCrosstalkSources_1031714_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0c106321-bdc2-476f-b29e-a77122025938_isrForCrosstalkSources_1031714_40.stderr parsl.dataflow.dflow INFO: Task 1480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4ec53eef3cf040d18f89214093c285f with result from task 1480 parsl.dataflow.dflow DEBUG: Task 1480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/26835edb-4046-4d6f-94eb-09fd3d22c754_isrForCrosstalkSources_1031718_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/26835edb-4046-4d6f-94eb-09fd3d22c754_isrForCrosstalkSources_1031718_6.stderr parsl.dataflow.dflow INFO: Task 1449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95739c3d7d5ce0ae41d002f94313cd3e with result from task 1449 parsl.dataflow.dflow DEBUG: Task 1449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/92b57920-0a32-4131-b47a-09ddee6aa8b1_isrForCrosstalkSources_1031716_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/92b57920-0a32-4131-b47a-09ddee6aa8b1_isrForCrosstalkSources_1031716_17.stderr parsl.dataflow.dflow INFO: Task 1631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9da319d8abfcf148575e00a14d19596 with result from task 1631 parsl.dataflow.dflow DEBUG: Task 1631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/97cdf895-34c4-423d-87cd-82343cfc22aa_isrForCrosstalkSources_1031729_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/97cdf895-34c4-423d-87cd-82343cfc22aa_isrForCrosstalkSources_1031729_23.stderr parsl.dataflow.dflow INFO: Task 1517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afcbce7f5733aad7bfc63a9423763ee1 with result from task 1517 parsl.dataflow.dflow DEBUG: Task 1517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/0f22c703-ccff-4b67-8561-7f29e1f59806_isrForCrosstalkSources_1031728_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/0f22c703-ccff-4b67-8561-7f29e1f59806_isrForCrosstalkSources_1031728_18.stderr parsl.dataflow.dflow INFO: Task 1530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5d11ac39bce7803baa108f6f4aa12e5 with result from task 1530 parsl.dataflow.dflow DEBUG: Task 1530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/b6a68dee-3122-433f-be2e-51812e72f94e_isrForCrosstalkSources_1031722_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/b6a68dee-3122-433f-be2e-51812e72f94e_isrForCrosstalkSources_1031722_39.stderr 2024-03-17 04:06:09 proc_lsst.multi:170 [INFO] [multi] job_id 17148097 proc_lsst.multi INFO: [multi] job_id 17148097 2024-03-17 04:06:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 96 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 96 2024-03-17 04:06:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148097 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148097 parsl.executors.status_handling DEBUG: Launched block 151 on executor multi with job ID 17148097 parsl.executors.status_handling INFO: Allocated block ID 152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-152 2024-03-17 04:06:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 parsl.dataflow.dflow INFO: Task 1538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5070536fade253a1af04b97bbe9d71f7 with result from task 1538 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 2024-03-17 04:06:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 96 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 96 2024-03-17 04:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 1538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/809fe97f-387a-4cd0-b140-a9be3a4f2ce6_isrForCrosstalkSources_1031713_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/809fe97f-387a-4cd0-b140-a9be3a4f2ce6_isrForCrosstalkSources_1031713_56.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ff9ed00baa3f7b2bfadd1fb5411a20a with result from task 1521 parsl.dataflow.dflow DEBUG: Task 1521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ece69d7f-d8cf-4582-8724-300cfa1286fe_isrForCrosstalkSources_1031715_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/ece69d7f-d8cf-4582-8724-300cfa1286fe_isrForCrosstalkSources_1031715_6.stderr parsl.dataflow.dflow INFO: Task 1633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e731e9240abf8ccc51fd1604deab620 with result from task 1633 parsl.dataflow.dflow DEBUG: Task 1633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/fb8848ba-c06d-4134-99ac-7058c88baadc_isrForCrosstalkSources_1031733_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/fb8848ba-c06d-4134-99ac-7058c88baadc_isrForCrosstalkSources_1031733_33.stderr parsl.dataflow.dflow INFO: Task 1175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b61bd93c8b9686a36058b8494eb48b2 with result from task 1175 parsl.dataflow.dflow DEBUG: Task 1175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a7869cf1-8ba4-471f-a0f8-ad250473f5c9_isrForCrosstalkSources_1031706_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a7869cf1-8ba4-471f-a0f8-ad250473f5c9_isrForCrosstalkSources_1031706_7.stderr parsl.dataflow.dflow INFO: Task 1289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67a213cc02fc4883f8af343807846c87 with result from task 1289 parsl.dataflow.dflow DEBUG: Task 1289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5a00368c-db7c-4e62-96ba-1e02fbcb1d2c_isrForCrosstalkSources_1031728_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5a00368c-db7c-4e62-96ba-1e02fbcb1d2c_isrForCrosstalkSources_1031728_47.stderr 2024-03-17 04:06:10 proc_lsst.multi:170 [INFO] [multi] job_id 17148098 proc_lsst.multi INFO: [multi] job_id 17148098 2024-03-17 04:06:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 97 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 97 2024-03-17 04:06:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148098 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148098 parsl.executors.status_handling DEBUG: Launched block 152 on executor multi with job ID 17148098 parsl.executors.status_handling INFO: Allocated block ID 153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-153 2024-03-17 04:06:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 2024-03-17 04:06:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 97 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 97 2024-03-17 04:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a99ad990634e2eb824a1f832dbd702bd with result from task 1686 parsl.dataflow.dflow DEBUG: Task 1686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/89621e6c-8956-43ce-93ee-891996465707_isrForCrosstalkSources_1031715_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/89621e6c-8956-43ce-93ee-891996465707_isrForCrosstalkSources_1031715_55.stderr parsl.dataflow.dflow INFO: Task 1194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df86ad577af0b62425ce74dda3756580 with result from task 1194 parsl.dataflow.dflow DEBUG: Task 1194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/59020227-c6db-4ac0-8b56-1f9d2aa8191f_isrForCrosstalkSources_1031727_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/59020227-c6db-4ac0-8b56-1f9d2aa8191f_isrForCrosstalkSources_1031727_49.stderr parsl.dataflow.dflow INFO: Task 1609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a31645cde6aa82bd5dbddcd37866c84 with result from task 1609 parsl.dataflow.dflow DEBUG: Task 1609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ab56e0a7-f44c-4fa3-baeb-7cd0fa645735_isrForCrosstalkSources_1031731_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ab56e0a7-f44c-4fa3-baeb-7cd0fa645735_isrForCrosstalkSources_1031731_14.stderr parsl.dataflow.dflow INFO: Task 1405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1f9e15dcd94734902a75d52caea10d5 with result from task 1405 parsl.dataflow.dflow DEBUG: Task 1405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/9da09d4a-a72d-4e8b-ab54-6df72bcb7d61_isrForCrosstalkSources_1031703_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/9da09d4a-a72d-4e8b-ab54-6df72bcb7d61_isrForCrosstalkSources_1031703_2.stderr parsl.dataflow.dflow INFO: Task 1555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72801248a6b8925c9ccbc3934c076649 with result from task 1555 parsl.dataflow.dflow DEBUG: Task 1555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/79d05077-02ea-4f92-bf67-799c3a9b3710_isrForCrosstalkSources_1031706_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/79d05077-02ea-4f92-bf67-799c3a9b3710_isrForCrosstalkSources_1031706_15.stderr parsl.dataflow.dflow INFO: Task 1466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d3768e4487ad91d3ab715b8cf99ccc8 with result from task 1466 parsl.dataflow.dflow DEBUG: Task 1466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a9485acb-61fe-4b6a-9b8c-53eae4d4571d_isrForCrosstalkSources_1031730_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/a9485acb-61fe-4b6a-9b8c-53eae4d4571d_isrForCrosstalkSources_1031730_18.stderr 2024-03-17 04:06:10 proc_lsst.multi:170 [INFO] [multi] job_id 17148099 proc_lsst.multi INFO: [multi] job_id 17148099 2024-03-17 04:06:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 98 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 98 2024-03-17 04:06:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148099 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148099 parsl.executors.status_handling DEBUG: Launched block 153 on executor multi with job ID 17148099 parsl.executors.status_handling INFO: Allocated block ID 154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-154 2024-03-17 04:06:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 2024-03-17 04:06:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 parsl.dataflow.dflow INFO: Task 1432 completed (launched -> exec_done) proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.memoization DEBUG: Storing app cache entry 3c2178945580a200f148cdd639fbda5a with result from task 1432 proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 98 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 98 2024-03-17 04:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 1432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4391da84-2813-4a68-9e2c-03439f0bfed2_isrForCrosstalkSources_1031715_31.stdout parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard error for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4391da84-2813-4a68-9e2c-03439f0bfed2_isrForCrosstalkSources_1031715_31.stderr parsl.dataflow.dflow INFO: Task 1532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9eee8d61a2f8b51c134a06d9bce1070c with result from task 1532 parsl.dataflow.dflow DEBUG: Task 1532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1baae9f5-c5a4-45f6-ba2b-6af00c6603bd_isrForCrosstalkSources_1031715_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1baae9f5-c5a4-45f6-ba2b-6af00c6603bd_isrForCrosstalkSources_1031715_21.stderr parsl.dataflow.dflow INFO: Task 1456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 436828e64d05470389b13d2380912ec8 with result from task 1456 parsl.dataflow.dflow DEBUG: Task 1456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/8ce63b30-89ee-4391-bfd9-76dccd77b5dc_isrForCrosstalkSources_1031711_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/8ce63b30-89ee-4391-bfd9-76dccd77b5dc_isrForCrosstalkSources_1031711_36.stderr parsl.dataflow.dflow INFO: Task 1548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 397eb72b0f3b788ab810cb992d02986c with result from task 1548 parsl.dataflow.dflow DEBUG: Task 1548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/1150f1c4-c624-46fb-ba69-d20ee4919bc3_isrForCrosstalkSources_1031726_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/1150f1c4-c624-46fb-ba69-d20ee4919bc3_isrForCrosstalkSources_1031726_40.stderr parsl.dataflow.dflow INFO: Task 1551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a8f3ab426d1ed53ceeed488beab2cd2 with result from task 1551 parsl.dataflow.dflow DEBUG: Task 1551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c33ed61e-0225-4b63-a1d6-2ce22456b85f_isrForCrosstalkSources_1031729_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c33ed61e-0225-4b63-a1d6-2ce22456b85f_isrForCrosstalkSources_1031729_51.stderr parsl.dataflow.dflow INFO: Task 1507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca2bfad7038b942e541a9e7f33a79cdd with result from task 1507 parsl.dataflow.dflow DEBUG: Task 1507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/77a0d118-e842-4d3d-ac3f-c894cf172911_isrForCrosstalkSources_1031708_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/77a0d118-e842-4d3d-ac3f-c894cf172911_isrForCrosstalkSources_1031708_2.stderr parsl.dataflow.dflow INFO: Task 1415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a409cbc48e7599ae0decff030afa959e with result from task 1415 parsl.dataflow.dflow DEBUG: Task 1415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/b39c5816-b6d9-49f3-b2f2-2329f2e00226_isrForCrosstalkSources_1031725_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/b39c5816-b6d9-49f3-b2f2-2329f2e00226_isrForCrosstalkSources_1031725_4.stderr parsl.dataflow.dflow INFO: Task 1699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65f9d2533a8fbe282bb106150f8747f3 with result from task 1699 parsl.dataflow.dflow DEBUG: Task 1699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/96c43b4a-8608-47b6-b586-3fc5031bd7ec_isrForCrosstalkSources_1031723_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/96c43b4a-8608-47b6-b586-3fc5031bd7ec_isrForCrosstalkSources_1031723_43.stderr parsl.dataflow.dflow INFO: Task 1552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 391fda5f7015db426b5a70121426e5d6 with result from task 1552 parsl.dataflow.dflow DEBUG: Task 1552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/19a6821c-ad6c-43dc-826f-749b7833b4d2_isrForCrosstalkSources_1031734_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/19a6821c-ad6c-43dc-826f-749b7833b4d2_isrForCrosstalkSources_1031734_11.stderr 2024-03-17 04:06:11 proc_lsst.multi:170 [INFO] [multi] job_id 17148100 proc_lsst.multi INFO: [multi] job_id 17148100 2024-03-17 04:06:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 99 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 99 2024-03-17 04:06:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148100 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148100 parsl.executors.status_handling DEBUG: Launched block 154 on executor multi with job ID 17148100 parsl.executors.status_handling INFO: Allocated block ID 155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-155 2024-03-17 04:06:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 2024-03-17 04:06:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 99 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 99 2024-03-17 04:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9501c5a1d7ec14e72588bd4e77dfb937 with result from task 1592 parsl.dataflow.dflow DEBUG: Task 1592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/11c4cb86-47a0-4c60-9447-bca87e3cb14a_isrForCrosstalkSources_1031714_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/11c4cb86-47a0-4c60-9447-bca87e3cb14a_isrForCrosstalkSources_1031714_51.stderr parsl.dataflow.dflow INFO: Task 1414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b16d3cae3b968c5e20ab84e29e496f2 with result from task 1414 parsl.dataflow.dflow DEBUG: Task 1414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a2f9b1ac-20f6-4083-996d-d113b7db4da4_isrForCrosstalkSources_1031701_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a2f9b1ac-20f6-4083-996d-d113b7db4da4_isrForCrosstalkSources_1031701_31.stderr parsl.dataflow.dflow INFO: Task 1404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 081b344f0797a4e4021c70e0b3846af4 with result from task 1404 parsl.dataflow.dflow DEBUG: Task 1404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/b8ac035f-ec70-4671-b847-89d8eb561e2a_isrForCrosstalkSources_1031705_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/b8ac035f-ec70-4671-b847-89d8eb561e2a_isrForCrosstalkSources_1031705_28.stderr parsl.dataflow.dflow INFO: Task 1189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e95fc30519d2cc50dc1d47be01252f87 with result from task 1189 parsl.dataflow.dflow DEBUG: Task 1189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/74b5fc25-5f4e-4557-b35b-25c97608c454_isrForCrosstalkSources_1031723_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/74b5fc25-5f4e-4557-b35b-25c97608c454_isrForCrosstalkSources_1031723_3.stderr parsl.dataflow.dflow INFO: Task 1531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90162d2be957d5aa2e1de994e2ee78b0 with result from task 1531 parsl.dataflow.dflow DEBUG: Task 1531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/59296da6-b6e4-4651-b94d-9b23d91080d6_isrForCrosstalkSources_1031700_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/59296da6-b6e4-4651-b94d-9b23d91080d6_isrForCrosstalkSources_1031700_24.stderr parsl.dataflow.dflow INFO: Task 1525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d72e798c46fe69fc239b844f6aad2029 with result from task 1525 parsl.dataflow.dflow DEBUG: Task 1525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/4895fd93-f0df-4313-9314-b40c583edab0_isrForCrosstalkSources_1031714_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/4895fd93-f0df-4313-9314-b40c583edab0_isrForCrosstalkSources_1031714_16.stderr 2024-03-17 04:06:12 proc_lsst.multi:170 [INFO] [multi] job_id 17148101 proc_lsst.multi INFO: [multi] job_id 17148101 2024-03-17 04:06:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 100 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 100 2024-03-17 04:06:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148101 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148101 parsl.executors.status_handling DEBUG: Launched block 155 on executor multi with job ID 17148101 parsl.executors.status_handling INFO: Allocated block ID 156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-156 2024-03-17 04:06:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 2024-03-17 04:06:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 100 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 100 2024-03-17 04:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 867619e452e416fc1c4d6159793edd47 with result from task 1705 parsl.dataflow.dflow DEBUG: Task 1705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8903fa01-6d48-44f9-b8c8-e534dd994de7_isrForCrosstalkSources_1031730_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8903fa01-6d48-44f9-b8c8-e534dd994de7_isrForCrosstalkSources_1031730_46.stderr parsl.dataflow.dflow INFO: Task 1560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53d8957e0e96daf83a43aa3129436970 with result from task 1560 parsl.dataflow.dflow DEBUG: Task 1560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/18d1e226-817c-4380-830e-3059cac397ca_isrForCrosstalkSources_1031700_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/18d1e226-817c-4380-830e-3059cac397ca_isrForCrosstalkSources_1031700_38.stderr parsl.dataflow.dflow INFO: Task 1509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 704a10a02628cdbe3144a6ac9d63789a with result from task 1509 parsl.dataflow.dflow DEBUG: Task 1509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8c9af191-f4e5-4e0e-957f-83eefe1a8295_isrForCrosstalkSources_1031700_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8c9af191-f4e5-4e0e-957f-83eefe1a8295_isrForCrosstalkSources_1031700_48.stderr parsl.dataflow.dflow INFO: Task 1540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 728c43fcfec46cd431fe49ca40101120 with result from task 1540 parsl.dataflow.dflow DEBUG: Task 1540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/575d2e01-eda0-4378-84e6-2e771a4237b5_isrForCrosstalkSources_1031706_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/575d2e01-eda0-4378-84e6-2e771a4237b5_isrForCrosstalkSources_1031706_51.stderr parsl.dataflow.dflow INFO: Task 1576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57c2d079ee874ee52a0bfa7f105252c9 with result from task 1576 parsl.dataflow.dflow DEBUG: Task 1576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/653cfeab-a6bc-4cbc-be9d-d692a8c4ae08_isrForCrosstalkSources_1031705_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/653cfeab-a6bc-4cbc-be9d-d692a8c4ae08_isrForCrosstalkSources_1031705_54.stderr 2024-03-17 04:06:12 proc_lsst.multi:170 [INFO] [multi] job_id 17148102 proc_lsst.multi INFO: [multi] job_id 17148102 2024-03-17 04:06:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 101 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 101 2024-03-17 04:06:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148102 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148102 parsl.executors.status_handling DEBUG: Launched block 156 on executor multi with job ID 17148102 parsl.executors.status_handling INFO: Allocated block ID 157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-157 2024-03-17 04:06:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 2024-03-17 04:06:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 101 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 101 2024-03-17 04:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 791306daf64d986eff09d5574b65bfdb with result from task 1390 parsl.dataflow.dflow DEBUG: Task 1390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/7e3d2aa4-ba2d-4fbb-9ca4-5307df8633c4_isrForCrosstalkSources_1031719_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/7e3d2aa4-ba2d-4fbb-9ca4-5307df8633c4_isrForCrosstalkSources_1031719_50.stderr parsl.dataflow.dflow INFO: Task 1499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ce5dc02519317445a9aaeca335d34cc with result from task 1499 parsl.dataflow.dflow DEBUG: Task 1499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/520d09e9-17e0-4f67-8d06-d6bfdde317b5_isrForCrosstalkSources_1031702_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/520d09e9-17e0-4f67-8d06-d6bfdde317b5_isrForCrosstalkSources_1031702_8.stderr parsl.dataflow.dflow INFO: Task 1454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc57689faf0ac703ba8bc5cb98327e09 with result from task 1454 parsl.dataflow.dflow DEBUG: Task 1454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/2ca39469-e244-4618-9d8d-9d9d7fc0c1b9_isrForCrosstalkSources_1031721_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/2ca39469-e244-4618-9d8d-9d9d7fc0c1b9_isrForCrosstalkSources_1031721_22.stderr parsl.dataflow.dflow INFO: Task 1537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e5ed4aa6eb49c15580412dd5f7dae1f with result from task 1537 parsl.dataflow.dflow DEBUG: Task 1537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/4b0ad832-fb26-409f-9c15-c4dd655cee7a_isrForCrosstalkSources_1031713_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/4b0ad832-fb26-409f-9c15-c4dd655cee7a_isrForCrosstalkSources_1031713_12.stderr parsl.dataflow.dflow INFO: Task 1526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ec9f8d446fc81a1953aa471747ffe57 with result from task 1526 parsl.dataflow.dflow DEBUG: Task 1526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/35fcb2a1-44a8-4d66-9f68-77ecf3330824_isrForCrosstalkSources_1031705_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/35fcb2a1-44a8-4d66-9f68-77ecf3330824_isrForCrosstalkSources_1031705_15.stderr parsl.dataflow.dflow INFO: Task 1757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a69152406c83b7ffac47da9b252b0dc with result from task 1757 parsl.dataflow.dflow DEBUG: Task 1757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/e18c0920-1090-4925-a249-d79d3bd5ad1f_isrForCrosstalkSources_1031729_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/e18c0920-1090-4925-a249-d79d3bd5ad1f_isrForCrosstalkSources_1031729_26.stderr parsl.dataflow.dflow INFO: Task 1564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48207b416ceb6fbb9417bd2bda9923b1 with result from task 1564 parsl.dataflow.dflow DEBUG: Task 1564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/bc694f37-1e14-4ba9-bf13-ca29db973667_isrForCrosstalkSources_1031702_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/bc694f37-1e14-4ba9-bf13-ca29db973667_isrForCrosstalkSources_1031702_18.stderr parsl.dataflow.dflow INFO: Task 1570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0d40a2cbfc796b5f71e779e21e14aeb with result from task 1570 parsl.dataflow.dflow DEBUG: Task 1570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/428b2cec-2717-485f-864e-171b1cfa90e5_isrForCrosstalkSources_1031728_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/428b2cec-2717-485f-864e-171b1cfa90e5_isrForCrosstalkSources_1031728_32.stderr parsl.dataflow.dflow INFO: Task 1455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c9c86eb0fa908463c00f1c830c6c87f with result from task 1455 parsl.dataflow.dflow DEBUG: Task 1455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/43e87c8a-6ddc-49c0-b38f-eafd387be476_isrForCrosstalkSources_1031734_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/43e87c8a-6ddc-49c0-b38f-eafd387be476_isrForCrosstalkSources_1031734_26.stderr parsl.dataflow.dflow INFO: Task 1566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6abe49e56ae36c0723d7f35779225d3 with result from task 1566 parsl.dataflow.dflow DEBUG: Task 1566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/01df4bf8-8390-4bfb-a97e-ac0d08ae740b_isrForCrosstalkSources_1031710_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/01df4bf8-8390-4bfb-a97e-ac0d08ae740b_isrForCrosstalkSources_1031710_17.stderr parsl.dataflow.dflow INFO: Task 1602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 044e9b68e753487496ff76755f74c314 with result from task 1602 parsl.dataflow.dflow DEBUG: Task 1602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/315aeffb-5760-400b-9933-d54d07a0f3f9_isrForCrosstalkSources_1031738_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/315aeffb-5760-400b-9933-d54d07a0f3f9_isrForCrosstalkSources_1031738_48.stderr parsl.dataflow.dflow INFO: Task 1447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffd0234fe173fa72303a53988ebf6747 with result from task 1447 parsl.dataflow.dflow DEBUG: Task 1447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/97cabd04-f316-4a3f-91ae-02167709003a_isrForCrosstalkSources_1031721_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/97cabd04-f316-4a3f-91ae-02167709003a_isrForCrosstalkSources_1031721_43.stderr parsl.dataflow.dflow INFO: Task 1433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6488faf8bdcb9816220f4fbbd45c6a6 with result from task 1433 parsl.dataflow.dflow DEBUG: Task 1433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/54d1b2f6-7cd4-4d39-91f9-5ee17b9e4849_isrForCrosstalkSources_1031718_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/54d1b2f6-7cd4-4d39-91f9-5ee17b9e4849_isrForCrosstalkSources_1031718_61.stderr parsl.dataflow.dflow INFO: Task 1460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf003393001f74cc35a305d5d08ecd2a with result from task 1460 parsl.dataflow.dflow DEBUG: Task 1460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/77feddb7-0442-4d96-88a1-7f04d59ac5bf_isrForCrosstalkSources_1031725_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/77feddb7-0442-4d96-88a1-7f04d59ac5bf_isrForCrosstalkSources_1031725_62.stderr parsl.dataflow.dflow INFO: Task 1515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1950f6c5b6e73a9d75201ebf53d8faf7 with result from task 1515 parsl.dataflow.dflow DEBUG: Task 1515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/dbe3aa3f-53af-4d0f-8dfd-31b9bf770a90_isrForCrosstalkSources_1031702_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/dbe3aa3f-53af-4d0f-8dfd-31b9bf770a90_isrForCrosstalkSources_1031702_22.stderr parsl.dataflow.dflow INFO: Task 1558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1512412fe917849ef3cf0b7bbe62963a with result from task 1558 parsl.dataflow.dflow DEBUG: Task 1558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bdcbb4b5-8645-494e-afd6-2c54c5f7e929_isrForCrosstalkSources_1031715_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bdcbb4b5-8645-494e-afd6-2c54c5f7e929_isrForCrosstalkSources_1031715_62.stderr parsl.dataflow.dflow INFO: Task 1462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d6390743375cadfc9d0bdd3bc2a5cfc with result from task 1462 parsl.dataflow.dflow DEBUG: Task 1462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/abc33104-5273-46a9-813e-7989f8905d1e_isrForCrosstalkSources_1031730_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/abc33104-5273-46a9-813e-7989f8905d1e_isrForCrosstalkSources_1031730_28.stderr parsl.dataflow.dflow INFO: Task 1400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 976637cb8c92457e0dd76a75a8dd6749 with result from task 1400 parsl.dataflow.dflow DEBUG: Task 1400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3e79db8c-38fa-462c-b23e-68f74a24406b_isrForCrosstalkSources_1031736_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3e79db8c-38fa-462c-b23e-68f74a24406b_isrForCrosstalkSources_1031736_60.stderr parsl.dataflow.dflow INFO: Task 1707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9630ed5001f0d54328e72b754911d952 with result from task 1707 parsl.dataflow.dflow DEBUG: Task 1707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d18c024a-4176-4d80-836c-7960cda4c9a5_isrForCrosstalkSources_1031731_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/d18c024a-4176-4d80-836c-7960cda4c9a5_isrForCrosstalkSources_1031731_3.stderr parsl.dataflow.dflow INFO: Task 1428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3410c309b43660e96aad0fcd0ca8d8e with result from task 1428 parsl.dataflow.dflow DEBUG: Task 1428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a5a8154f-b128-4bbe-8d9e-846022eaf3f3_isrForCrosstalkSources_1031709_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a5a8154f-b128-4bbe-8d9e-846022eaf3f3_isrForCrosstalkSources_1031709_6.stderr parsl.dataflow.dflow INFO: Task 1714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6daf31c845edead59b786d56be0ad70 with result from task 1714 parsl.dataflow.dflow DEBUG: Task 1714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f814af3e-4b99-490a-8be0-49ee9bd00d56_isrForCrosstalkSources_1031720_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/f814af3e-4b99-490a-8be0-49ee9bd00d56_isrForCrosstalkSources_1031720_4.stderr parsl.dataflow.dflow INFO: Task 1577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5de3236a04ed228f21c3f56a4a24fa74 with result from task 1577 parsl.dataflow.dflow DEBUG: Task 1577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/3b71408a-28ff-450d-9e2e-f14ddefbab91_isrForCrosstalkSources_1031726_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/3b71408a-28ff-450d-9e2e-f14ddefbab91_isrForCrosstalkSources_1031726_18.stderr parsl.dataflow.dflow INFO: Task 1626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f03153529810575b7e2d7873f5683cb with result from task 1626 parsl.dataflow.dflow DEBUG: Task 1626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/7f85936d-6536-4baf-9e96-be35d823971a_isrForCrosstalkSources_1031726_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/7f85936d-6536-4baf-9e96-be35d823971a_isrForCrosstalkSources_1031726_27.stderr parsl.dataflow.dflow INFO: Task 1770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96a00502b27a7023f7d423f4fc119d39 with result from task 1770 parsl.dataflow.dflow DEBUG: Task 1770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/b664984b-c42a-4df6-a5b2-1fa2d11706cb_isrForCrosstalkSources_1031715_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/b664984b-c42a-4df6-a5b2-1fa2d11706cb_isrForCrosstalkSources_1031715_54.stderr parsl.dataflow.dflow INFO: Task 1416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2965db6b3dbcd01289c0eab81b82b7f with result from task 1416 parsl.dataflow.dflow DEBUG: Task 1416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/26882130-1bd0-41b8-a8a8-e59a6c4bc422_isrForCrosstalkSources_1031722_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/26882130-1bd0-41b8-a8a8-e59a6c4bc422_isrForCrosstalkSources_1031722_58.stderr parsl.dataflow.dflow INFO: Task 1464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fab2d1ad33745f9c12eb7dd0b5cb2da8 with result from task 1464 parsl.dataflow.dflow DEBUG: Task 1464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/10c7bf26-7c2b-4fa7-9b18-9aba35e43558_isrForCrosstalkSources_1031700_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/10c7bf26-7c2b-4fa7-9b18-9aba35e43558_isrForCrosstalkSources_1031700_55.stderr parsl.dataflow.dflow INFO: Task 1778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13a1ce16fb6c6cbd8bc1a277f41de1f3 with result from task 1778 parsl.dataflow.dflow DEBUG: Task 1778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/987c701d-ecf9-4cf1-bba4-0a8a6ad84f17_isrForCrosstalkSources_1031736_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/987c701d-ecf9-4cf1-bba4-0a8a6ad84f17_isrForCrosstalkSources_1031736_56.stderr parsl.dataflow.dflow INFO: Task 1773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ae185745ba2da3b2dab2c1bf4a71d4d with result from task 1773 parsl.dataflow.dflow DEBUG: Task 1773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6afc3bff-7c9a-4031-8023-24dfe24ff50f_isrForCrosstalkSources_1031737_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6afc3bff-7c9a-4031-8023-24dfe24ff50f_isrForCrosstalkSources_1031737_45.stderr parsl.dataflow.dflow INFO: Task 1513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c1113079e91fcd2f96a332a6358a3f2 with result from task 1513 parsl.dataflow.dflow DEBUG: Task 1513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/403a7066-fe8d-426b-812a-3716b188bfd4_isrForCrosstalkSources_1031729_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/403a7066-fe8d-426b-812a-3716b188bfd4_isrForCrosstalkSources_1031729_1.stderr parsl.dataflow.dflow INFO: Task 1767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42d4104c947d482f5cf8a49491d4e132 with result from task 1767 parsl.dataflow.dflow DEBUG: Task 1767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/63f55131-6da6-4d2d-a5b5-4473f5d98d20_isrForCrosstalkSources_1031712_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/63f55131-6da6-4d2d-a5b5-4473f5d98d20_isrForCrosstalkSources_1031712_32.stderr parsl.dataflow.dflow INFO: Task 1666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e366dcaf6b7729930c844a6c2900e35f with result from task 1666 parsl.dataflow.dflow DEBUG: Task 1666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/da929f66-7228-433a-9ef0-8df9e6bddb59_isrForCrosstalkSources_1031736_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/da929f66-7228-433a-9ef0-8df9e6bddb59_isrForCrosstalkSources_1031736_32.stderr parsl.dataflow.dflow INFO: Task 1425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9191fd641f2dea1eabe48f9b43e111c9 with result from task 1425 parsl.dataflow.dflow DEBUG: Task 1425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bab9a51e-c698-4cf4-8e6d-324a4650aa39_isrForCrosstalkSources_1031734_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/bab9a51e-c698-4cf4-8e6d-324a4650aa39_isrForCrosstalkSources_1031734_47.stderr parsl.dataflow.dflow INFO: Task 1122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6395e67214033595ac82da1daf74931 with result from task 1122 parsl.dataflow.dflow DEBUG: Task 1122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/6b220235-8001-4299-b9c5-bb6f706eb970_isrForCrosstalkSources_1031714_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/6b220235-8001-4299-b9c5-bb6f706eb970_isrForCrosstalkSources_1031714_59.stderr parsl.dataflow.dflow INFO: Task 1545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa5f0df45b1ff8a8904e7524db68508 with result from task 1545 parsl.dataflow.dflow DEBUG: Task 1545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b907b494-1fee-449f-bea0-e8d761472e77_isrForCrosstalkSources_1031713_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b907b494-1fee-449f-bea0-e8d761472e77_isrForCrosstalkSources_1031713_1.stderr parsl.dataflow.dflow INFO: Task 1694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e3ee2cf6e5d37be272492f31eb8399b with result from task 1694 parsl.dataflow.dflow DEBUG: Task 1694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1508f187-6b74-4126-beec-a669866aac7a_isrForCrosstalkSources_1031700_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1508f187-6b74-4126-beec-a669866aac7a_isrForCrosstalkSources_1031700_11.stderr parsl.dataflow.dflow INFO: Task 1747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92a5a15a3e82905e1f0c06cfd44d48d8 with result from task 1747 parsl.dataflow.dflow DEBUG: Task 1747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c8e4ebab-dd23-497f-b28b-af761c66ea32_isrForCrosstalkSources_1031734_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c8e4ebab-dd23-497f-b28b-af761c66ea32_isrForCrosstalkSources_1031734_33.stderr parsl.dataflow.dflow INFO: Task 1739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01a6580e5f0ef7e6976186723f5ff842 with result from task 1739 parsl.dataflow.dflow DEBUG: Task 1739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5341bff8-6a54-48d7-be86-605dbf804f75_isrForCrosstalkSources_1031701_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5341bff8-6a54-48d7-be86-605dbf804f75_isrForCrosstalkSources_1031701_55.stderr parsl.dataflow.dflow INFO: Task 1529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b0cb9b7661a3d644d086fc03f33c76f with result from task 1529 parsl.dataflow.dflow DEBUG: Task 1529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/2b7f4856-9bdc-4941-8ae6-2be2b761e961_isrForCrosstalkSources_1031706_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/2b7f4856-9bdc-4941-8ae6-2be2b761e961_isrForCrosstalkSources_1031706_38.stderr parsl.dataflow.dflow INFO: Task 1479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba2d79396f5cfb3d7d733bc6d9a0e202 with result from task 1479 parsl.dataflow.dflow DEBUG: Task 1479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f3999668-55a4-40ed-b808-fc28ae52eb2f_isrForCrosstalkSources_1031699_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f3999668-55a4-40ed-b808-fc28ae52eb2f_isrForCrosstalkSources_1031699_29.stderr parsl.dataflow.dflow INFO: Task 1541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b41868b11a90f3b0d1d020d2fdf6115a with result from task 1541 parsl.dataflow.dflow DEBUG: Task 1541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d9133e78-0dc3-4f08-a343-4c81a0a6db1a_isrForCrosstalkSources_1031701_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d9133e78-0dc3-4f08-a343-4c81a0a6db1a_isrForCrosstalkSources_1031701_36.stderr parsl.dataflow.dflow INFO: Task 1422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85d8e4b624a63b5ac0eafe096ce855fe with result from task 1422 parsl.dataflow.dflow DEBUG: Task 1422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d0a28a4a-2bad-49d1-9028-3341a8a1a33a_isrForCrosstalkSources_1031722_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d0a28a4a-2bad-49d1-9028-3341a8a1a33a_isrForCrosstalkSources_1031722_52.stderr parsl.dataflow.dflow INFO: Task 1212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38753bb3175ba30959e6cd1e68134cf7 with result from task 1212 parsl.dataflow.dflow DEBUG: Task 1212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/a3fb240a-714c-4477-af1d-d2d348664b92_isrForCrosstalkSources_1031699_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/a3fb240a-714c-4477-af1d-d2d348664b92_isrForCrosstalkSources_1031699_7.stderr parsl.dataflow.dflow INFO: Task 1493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb0a53d4e5c426b2bd702bba020a941f with result from task 1493 parsl.dataflow.dflow DEBUG: Task 1493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/99d3a6d3-6343-421e-8e35-f313f9dd41e4_isrForCrosstalkSources_1031731_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/99d3a6d3-6343-421e-8e35-f313f9dd41e4_isrForCrosstalkSources_1031731_57.stderr parsl.dataflow.dflow INFO: Task 1632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88a1bed341a3581129e46d375f854af3 with result from task 1632 parsl.dataflow.dflow DEBUG: Task 1632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/4e97a336-9b60-4d73-94e4-11c6e1f2690e_isrForCrosstalkSources_1031726_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/4e97a336-9b60-4d73-94e4-11c6e1f2690e_isrForCrosstalkSources_1031726_6.stderr parsl.dataflow.dflow INFO: Task 1542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89432aa354802a0c5b5a42631bf93911 with result from task 1542 parsl.dataflow.dflow DEBUG: Task 1542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/efb3f015-224c-443f-a591-12521e63315d_isrForCrosstalkSources_1031705_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/efb3f015-224c-443f-a591-12521e63315d_isrForCrosstalkSources_1031705_47.stderr parsl.dataflow.dflow INFO: Task 1470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 444ae032a296c089d8ee576c54920f5d with result from task 1470 parsl.dataflow.dflow DEBUG: Task 1470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/1c40de5d-6e3e-4686-ad59-f95b9653de83_isrForCrosstalkSources_1031729_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/1c40de5d-6e3e-4686-ad59-f95b9653de83_isrForCrosstalkSources_1031729_57.stderr parsl.dataflow.dflow INFO: Task 1645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 695698ac990c400f870ad161f84804b3 with result from task 1645 parsl.dataflow.dflow DEBUG: Task 1645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5b41415a-0d89-40be-a789-a843bdfc6ee1_isrForCrosstalkSources_1031736_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/5b41415a-0d89-40be-a789-a843bdfc6ee1_isrForCrosstalkSources_1031736_45.stderr parsl.dataflow.dflow INFO: Task 1124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1679691fcca417189d15412d7fdfc041 with result from task 1124 parsl.dataflow.dflow DEBUG: Task 1124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0589e46d-f053-4acf-958d-d9cf151497dc_isrForCrosstalkSources_1031737_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0589e46d-f053-4acf-958d-d9cf151497dc_isrForCrosstalkSources_1031737_57.stderr parsl.dataflow.dflow INFO: Task 1413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 580900e091415549cdb163caa0bef00c with result from task 1413 parsl.dataflow.dflow DEBUG: Task 1413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/6ad70a18-0e19-428b-bf71-9103ba309d74_isrForCrosstalkSources_1031712_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/6ad70a18-0e19-428b-bf71-9103ba309d74_isrForCrosstalkSources_1031712_43.stderr parsl.dataflow.dflow INFO: Task 1116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d03010790b8030b64840e22d84f6ac82 with result from task 1116 parsl.dataflow.dflow DEBUG: Task 1116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a7c8ac75-b9d9-4abf-a9e4-944433eb7817_isrForCrosstalkSources_1031726_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/a7c8ac75-b9d9-4abf-a9e4-944433eb7817_isrForCrosstalkSources_1031726_60.stderr parsl.dataflow.dflow INFO: Task 1635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28b4fa38860bd78e2b5f295ae63dc954 with result from task 1635 parsl.dataflow.dflow DEBUG: Task 1635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5bf7ee2f-1e04-4b2c-a85e-2e123cdc3b50_isrForCrosstalkSources_1031711_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5bf7ee2f-1e04-4b2c-a85e-2e123cdc3b50_isrForCrosstalkSources_1031711_13.stderr parsl.dataflow.dflow INFO: Task 1637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f4a7c7d3f86f9136dec8c196abec117 with result from task 1637 parsl.dataflow.dflow DEBUG: Task 1637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/1e6ddc95-fbaf-4fcd-b97e-b656937d7661_isrForCrosstalkSources_1031720_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/1e6ddc95-fbaf-4fcd-b97e-b656937d7661_isrForCrosstalkSources_1031720_37.stderr parsl.dataflow.dflow INFO: Task 1641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c5c2df130b6d11f0db0ebcc4dddbcb4 with result from task 1641 parsl.dataflow.dflow DEBUG: Task 1641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3b58273c-42bb-4a13-9e37-b262cb61043c_isrForCrosstalkSources_1031723_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3b58273c-42bb-4a13-9e37-b262cb61043c_isrForCrosstalkSources_1031723_6.stderr parsl.dataflow.dflow INFO: Task 1496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7be1b12c321502826395fc2a3c63597e with result from task 1496 parsl.dataflow.dflow DEBUG: Task 1496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/26af4dbd-2472-4fb1-b046-3166fa94756a_isrForCrosstalkSources_1031707_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/26af4dbd-2472-4fb1-b046-3166fa94756a_isrForCrosstalkSources_1031707_23.stderr parsl.dataflow.dflow INFO: Task 1177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3560587fd79e7b8368d74c84027d2b6e with result from task 1177 parsl.dataflow.dflow DEBUG: Task 1177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/797b01fd-033d-4a9b-a966-50a04090d9f8_isrForCrosstalkSources_1031736_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/797b01fd-033d-4a9b-a966-50a04090d9f8_isrForCrosstalkSources_1031736_41.stderr parsl.dataflow.dflow INFO: Task 1733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29f5260ab844f57ff8fdaa10a476225f with result from task 1733 parsl.dataflow.dflow DEBUG: Task 1733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4bc7d57f-7bfd-4eab-b49c-4e31dfed20c2_isrForCrosstalkSources_1031715_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4bc7d57f-7bfd-4eab-b49c-4e31dfed20c2_isrForCrosstalkSources_1031715_7.stderr parsl.dataflow.dflow INFO: Task 1644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1470af92f1aff73868cd38e1fb4da520 with result from task 1644 parsl.dataflow.dflow DEBUG: Task 1644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b4d04676-b11e-448a-83a8-8e0f41801068_isrForCrosstalkSources_1031718_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b4d04676-b11e-448a-83a8-8e0f41801068_isrForCrosstalkSources_1031718_58.stderr parsl.dataflow.dflow INFO: Task 1217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3172f4a07d6498f7c5f803d111f2db6 with result from task 1217 parsl.dataflow.dflow DEBUG: Task 1217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/615dfcfd-f669-4e7a-a048-ec899d01fb99_isrForCrosstalkSources_1031717_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/615dfcfd-f669-4e7a-a048-ec899d01fb99_isrForCrosstalkSources_1031717_41.stderr parsl.dataflow.dflow INFO: Task 1623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 165aef1c8ed8831e8b85d8dc995e8283 with result from task 1623 parsl.dataflow.dflow DEBUG: Task 1623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/acc1af7a-b64a-4963-8e65-5d21b1d5696c_isrForCrosstalkSources_1031707_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/acc1af7a-b64a-4963-8e65-5d21b1d5696c_isrForCrosstalkSources_1031707_4.stderr parsl.dataflow.dflow INFO: Task 1781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce2ff8b9f644a42d502a339cc39e0a87 with result from task 1781 parsl.dataflow.dflow DEBUG: Task 1781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/fc50c121-3635-4a25-8468-ea7b99f22563_isrForCrosstalkSources_1031711_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/fc50c121-3635-4a25-8468-ea7b99f22563_isrForCrosstalkSources_1031711_49.stderr parsl.dataflow.dflow INFO: Task 1291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf7833a104e901e0ae5679160369362d with result from task 1291 parsl.dataflow.dflow DEBUG: Task 1291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f2ab1b01-0216-49eb-965f-fbd71be200ce_isrForCrosstalkSources_1031727_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f2ab1b01-0216-49eb-965f-fbd71be200ce_isrForCrosstalkSources_1031727_53.stderr parsl.dataflow.dflow INFO: Task 1320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e899c9361fa32992c52f77ba67658f87 with result from task 1320 parsl.dataflow.dflow DEBUG: Task 1320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/15100d57-cc45-4cbf-aa8d-204ed17caaba_isrForCrosstalkSources_1031727_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/15100d57-cc45-4cbf-aa8d-204ed17caaba_isrForCrosstalkSources_1031727_13.stderr parsl.dataflow.dflow INFO: Task 1753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf8f24d214201ffbc2507c4901f61a3 with result from task 1753 parsl.dataflow.dflow DEBUG: Task 1753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/5e549312-5c47-4fe7-8b05-b25a808d7722_isrForCrosstalkSources_1031719_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/5e549312-5c47-4fe7-8b05-b25a808d7722_isrForCrosstalkSources_1031719_57.stderr parsl.dataflow.dflow INFO: Task 1792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b47231b32200b98e767e3f670f182b41 with result from task 1792 parsl.dataflow.dflow DEBUG: Task 1792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/76fbcedc-9031-4374-ad9e-0d36c9d29b37_isrForCrosstalkSources_1031700_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/76fbcedc-9031-4374-ad9e-0d36c9d29b37_isrForCrosstalkSources_1031700_61.stderr parsl.dataflow.dflow INFO: Task 1784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41adb23c68eeed9ea9ed95b1f661f8fc with result from task 1784 parsl.dataflow.dflow DEBUG: Task 1784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/ad8db44d-42b4-46df-930f-7dca185bdfbc_isrForCrosstalkSources_1031736_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/ad8db44d-42b4-46df-930f-7dca185bdfbc_isrForCrosstalkSources_1031736_31.stderr parsl.dataflow.dflow INFO: Task 1434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bb4cb17e4de7aa975b9058b2f5b97f1 with result from task 1434 parsl.dataflow.dflow DEBUG: Task 1434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/809d18d7-ccf1-43e5-9520-e263abde32de_isrForCrosstalkSources_1031731_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/809d18d7-ccf1-43e5-9520-e263abde32de_isrForCrosstalkSources_1031731_11.stderr parsl.dataflow.dflow INFO: Task 1465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64afd691656bcc781d5d6635f0db9262 with result from task 1465 parsl.dataflow.dflow DEBUG: Task 1465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/4bd019b6-c100-45de-bff9-eddadf85a392_isrForCrosstalkSources_1031708_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/4bd019b6-c100-45de-bff9-eddadf85a392_isrForCrosstalkSources_1031708_62.stderr parsl.dataflow.dflow INFO: Task 1186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bab51d9ef72f0c9410f8d42088d327b5 with result from task 1186 parsl.dataflow.dflow DEBUG: Task 1186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/836afdc6-28c3-4d5b-b192-909535e0e424_isrForCrosstalkSources_1031711_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/836afdc6-28c3-4d5b-b192-909535e0e424_isrForCrosstalkSources_1031711_53.stderr parsl.dataflow.dflow INFO: Task 1575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2343df2d178038dde46ce35acb98b88 with result from task 1575 parsl.dataflow.dflow DEBUG: Task 1575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/e3261476-9ed2-4694-84bb-6dc6395fff2a_isrForCrosstalkSources_1031714_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/e3261476-9ed2-4694-84bb-6dc6395fff2a_isrForCrosstalkSources_1031714_24.stderr parsl.dataflow.dflow INFO: Task 1563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcc5ba025a4cef40183d4a90d611402c with result from task 1563 parsl.dataflow.dflow DEBUG: Task 1563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/2c5c49af-7958-41d9-992d-cc91da50250a_isrForCrosstalkSources_1031733_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/2c5c49af-7958-41d9-992d-cc91da50250a_isrForCrosstalkSources_1031733_60.stderr parsl.dataflow.dflow INFO: Task 1318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e89a51361f7c26af200d61ed65508a7a with result from task 1318 parsl.dataflow.dflow DEBUG: Task 1318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0222a112-71e1-40bb-aa7b-d658cdbc2adf_isrForCrosstalkSources_1031705_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0222a112-71e1-40bb-aa7b-d658cdbc2adf_isrForCrosstalkSources_1031705_33.stderr parsl.dataflow.dflow INFO: Task 1323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47a809405525b7d6cb707cd91a07805e with result from task 1323 parsl.dataflow.dflow DEBUG: Task 1323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d00c3c69-3697-42b4-b6c0-24bad7053af8_isrForCrosstalkSources_1031733_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d00c3c69-3697-42b4-b6c0-24bad7053af8_isrForCrosstalkSources_1031733_21.stderr parsl.dataflow.dflow INFO: Task 1846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35d419161389d924281bfc0784767264 with result from task 1846 parsl.dataflow.dflow DEBUG: Task 1846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/dfdc2043-c46d-41cf-a227-480cfd354901_isrForCrosstalkSources_1031700_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/dfdc2043-c46d-41cf-a227-480cfd354901_isrForCrosstalkSources_1031700_25.stderr parsl.dataflow.dflow INFO: Task 1547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 085922a7f33a26f3d6d2df80b994a231 with result from task 1547 parsl.dataflow.dflow DEBUG: Task 1547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/90172052-4a96-4c17-b7d0-278faa3f12c4_isrForCrosstalkSources_1031709_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/90172052-4a96-4c17-b7d0-278faa3f12c4_isrForCrosstalkSources_1031709_38.stderr 2024-03-17 04:06:19 proc_lsst.multi:170 [INFO] [multi] job_id 17148103 proc_lsst.multi INFO: [multi] job_id 17148103 2024-03-17 04:06:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 102 2024-03-17 04:06:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148103 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148103 parsl.executors.status_handling DEBUG: Launched block 157 on executor multi with job ID 17148103 parsl.executors.status_handling INFO: Allocated block ID 158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-158 2024-03-17 04:06:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 102 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c376cee28bb5972782c042b133c178aa with result from task 1522 parsl.dataflow.dflow DEBUG: Task 1522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/960b7339-9673-45cd-839d-ff4dee13a4c6_isrForCrosstalkSources_1031717_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/960b7339-9673-45cd-839d-ff4dee13a4c6_isrForCrosstalkSources_1031717_31.stderr parsl.dataflow.dflow INFO: Task 1556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98dc3fd33b2f10fb29730e180afad048 with result from task 1556 parsl.dataflow.dflow DEBUG: Task 1556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ea9a4594-3615-4341-a9e5-99742539997e_isrForCrosstalkSources_1031725_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/ea9a4594-3615-4341-a9e5-99742539997e_isrForCrosstalkSources_1031725_41.stderr 2024-03-17 04:06:19 proc_lsst.multi:170 [INFO] [multi] job_id 17148104 proc_lsst.multi INFO: [multi] job_id 17148104 2024-03-17 04:06:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 103 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 103 2024-03-17 04:06:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148104 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148104 parsl.executors.status_handling DEBUG: Launched block 158 on executor multi with job ID 17148104 parsl.executors.status_handling INFO: Allocated block ID 159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-159 2024-03-17 04:06:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 103 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 103 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15080d86dbc11486af25912dd8085506 with result from task 1562 parsl.dataflow.dflow DEBUG: Task 1562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8bed08f3-f3bb-43ef-b68d-07d5dd74743d_isrForCrosstalkSources_1031708_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8bed08f3-f3bb-43ef-b68d-07d5dd74743d_isrForCrosstalkSources_1031708_58.stderr parsl.dataflow.dflow INFO: Task 1427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a988e2f845889a07f3a629dc55daaa5a with result from task 1427 parsl.dataflow.dflow DEBUG: Task 1427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/981fd569-5b5c-4d72-a8b1-750657d4a653_isrForCrosstalkSources_1031736_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/981fd569-5b5c-4d72-a8b1-750657d4a653_isrForCrosstalkSources_1031736_53.stderr parsl.dataflow.dflow INFO: Task 1518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acde16198ec6b45afbf8deff62d26c68 with result from task 1518 parsl.dataflow.dflow DEBUG: Task 1518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f947840d-d6bd-4605-9cd9-54989176a2dc_isrForCrosstalkSources_1031727_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/f947840d-d6bd-4605-9cd9-54989176a2dc_isrForCrosstalkSources_1031727_50.stderr 2024-03-17 04:06:19 proc_lsst.multi:170 [INFO] [multi] job_id 17148105 proc_lsst.multi INFO: [multi] job_id 17148105 2024-03-17 04:06:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 104 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 104 2024-03-17 04:06:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148105 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148105 parsl.executors.status_handling DEBUG: Launched block 159 on executor multi with job ID 17148105 parsl.executors.status_handling INFO: Allocated block ID 160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-160 2024-03-17 04:06:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 104 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 104 2024-03-17 04:06:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a374829fb4ca690c2bb95853238d8b5a with result from task 1587 parsl.dataflow.dflow DEBUG: Task 1587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/38d2e4a8-e407-4752-aa5f-cd105b567997_isrForCrosstalkSources_1031718_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/38d2e4a8-e407-4752-aa5f-cd105b567997_isrForCrosstalkSources_1031718_60.stderr parsl.dataflow.dflow INFO: Task 1684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 179b85de61a05a49394043c0cdcf91b4 with result from task 1684 parsl.dataflow.dflow DEBUG: Task 1684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/4108829d-dec0-4bd3-9e6a-0de7fe6bb6ed_isrForCrosstalkSources_1031701_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/4108829d-dec0-4bd3-9e6a-0de7fe6bb6ed_isrForCrosstalkSources_1031701_45.stderr parsl.dataflow.dflow INFO: Task 1533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9075a6803a9e54d1505fe30c5fb52bc4 with result from task 1533 parsl.dataflow.dflow DEBUG: Task 1533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/ff0d99a6-8fb4-41f9-a7ac-115695794376_isrForCrosstalkSources_1031718_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/ff0d99a6-8fb4-41f9-a7ac-115695794376_isrForCrosstalkSources_1031718_50.stderr parsl.dataflow.dflow INFO: Task 1451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c061adbf72c0336d261083751369f75b with result from task 1451 parsl.dataflow.dflow DEBUG: Task 1451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/68542543-df2f-4c53-b9ee-b5ba5d55cfbf_isrForCrosstalkSources_1031703_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/68542543-df2f-4c53-b9ee-b5ba5d55cfbf_isrForCrosstalkSources_1031703_22.stderr parsl.dataflow.dflow INFO: Task 1418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9446ef4ce4c247d12314a0a6b9a8888 with result from task 1418 parsl.dataflow.dflow DEBUG: Task 1418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cd733fd3-9d76-4f7d-8c84-c9487da135cb_isrForCrosstalkSources_1031710_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cd733fd3-9d76-4f7d-8c84-c9487da135cb_isrForCrosstalkSources_1031710_3.stderr 2024-03-17 04:06:20 proc_lsst.multi:170 [INFO] [multi] job_id 17148106 proc_lsst.multi INFO: [multi] job_id 17148106 2024-03-17 04:06:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 105 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 105 2024-03-17 04:06:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148106 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148106 parsl.executors.status_handling DEBUG: Launched block 160 on executor multi with job ID 17148106 parsl.executors.status_handling INFO: Allocated block ID 161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-161 2024-03-17 04:06:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 105 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 105 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aae63be389d4a130b932c77064c12bce with result from task 1505 parsl.dataflow.dflow DEBUG: Task 1505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/cd4440af-7734-40b8-a177-8c66090fed1c_isrForCrosstalkSources_1031724_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/cd4440af-7734-40b8-a177-8c66090fed1c_isrForCrosstalkSources_1031724_53.stderr parsl.dataflow.dflow INFO: Task 1247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 169fa3fc0e235d2598e3c9df8d8d15d1 with result from task 1247 parsl.dataflow.dflow DEBUG: Task 1247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/50a48e88-c86b-4b31-bfa7-c3b775059636_isrForCrosstalkSources_1031701_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/50a48e88-c86b-4b31-bfa7-c3b775059636_isrForCrosstalkSources_1031701_19.stderr parsl.dataflow.dflow INFO: Task 1117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1deafd186cceb0a61eccac84752644ba with result from task 1117 parsl.dataflow.dflow DEBUG: Task 1117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/af3f3e77-1d4f-41d5-8491-92ae6c597f68_isrForCrosstalkSources_1031710_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/af3f3e77-1d4f-41d5-8491-92ae6c597f68_isrForCrosstalkSources_1031710_60.stderr parsl.dataflow.dflow INFO: Task 1261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c914e4e12afc8952b82b1a82ff61f18 with result from task 1261 parsl.dataflow.dflow DEBUG: Task 1261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d41dcbf0-f816-48ee-b26d-b51bed5a6863_isrForCrosstalkSources_1031701_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d41dcbf0-f816-48ee-b26d-b51bed5a6863_isrForCrosstalkSources_1031701_56.stderr 2024-03-17 04:06:20 proc_lsst.multi:170 [INFO] [multi] job_id 17148107 proc_lsst.multi INFO: [multi] job_id 17148107 2024-03-17 04:06:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 106 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 106 2024-03-17 04:06:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148107 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148107 parsl.executors.status_handling DEBUG: Launched block 161 on executor multi with job ID 17148107 parsl.executors.status_handling INFO: Allocated block ID 162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-162 2024-03-17 04:06:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 106 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 106 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9da2bce2e9639284e222778e6e79c600 with result from task 1539 parsl.dataflow.dflow DEBUG: Task 1539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/9b460b07-fd63-4dce-ac1a-733c752064f0_isrForCrosstalkSources_1031734_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/9b460b07-fd63-4dce-ac1a-733c752064f0_isrForCrosstalkSources_1031734_58.stderr parsl.dataflow.dflow INFO: Task 1246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 780d0b50887a6cf7c1bb94e0e59048d6 with result from task 1246 parsl.dataflow.dflow DEBUG: Task 1246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/aa7b5dec-77be-4027-a3d9-3b28d3e1ee71_isrForCrosstalkSources_1031730_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/aa7b5dec-77be-4027-a3d9-3b28d3e1ee71_isrForCrosstalkSources_1031730_19.stderr parsl.dataflow.dflow INFO: Task 1250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0d4b0d2b0216df22b595610611210ae with result from task 1250 parsl.dataflow.dflow DEBUG: Task 1250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/be7d6871-e4bf-4d27-8dce-79fc643fa67a_isrForCrosstalkSources_1031737_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/be7d6871-e4bf-4d27-8dce-79fc643fa67a_isrForCrosstalkSources_1031737_26.stderr parsl.dataflow.dflow INFO: Task 1260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 413d2e1bd95f949807bf4b8749b2e0f7 with result from task 1260 parsl.dataflow.dflow DEBUG: Task 1260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f651433c-de8e-40a7-b03c-8166a2108a8f_isrForCrosstalkSources_1031706_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f651433c-de8e-40a7-b03c-8166a2108a8f_isrForCrosstalkSources_1031706_25.stderr parsl.dataflow.dflow INFO: Task 1544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a2804937af00ed261dc3ac4a50064dd with result from task 1544 parsl.dataflow.dflow DEBUG: Task 1544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ac0995b0-e808-4b5b-9ae2-cb6aa0925897_isrForCrosstalkSources_1031726_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ac0995b0-e808-4b5b-9ae2-cb6aa0925897_isrForCrosstalkSources_1031726_50.stderr parsl.dataflow.dflow INFO: Task 1253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81728d7c640b69fab18ef9d680ac1073 with result from task 1253 parsl.dataflow.dflow DEBUG: Task 1253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/21b695ec-dc36-4bb9-b43f-ec7270461223_isrForCrosstalkSources_1031707_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/21b695ec-dc36-4bb9-b43f-ec7270461223_isrForCrosstalkSources_1031707_24.stderr parsl.dataflow.dflow INFO: Task 1184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f2bcfa1b2e45754b413f6cf7268a55a with result from task 1184 parsl.dataflow.dflow DEBUG: Task 1184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/99aabce6-4c08-4a98-9f78-1c12ca2bda2d_isrForCrosstalkSources_1031716_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/99aabce6-4c08-4a98-9f78-1c12ca2bda2d_isrForCrosstalkSources_1031716_47.stderr 2024-03-17 04:06:20 proc_lsst.multi:170 [INFO] [multi] job_id 17148108 proc_lsst.multi INFO: [multi] job_id 17148108 2024-03-17 04:06:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 107 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 107 2024-03-17 04:06:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148108 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148108 parsl.executors.status_handling DEBUG: Launched block 162 on executor multi with job ID 17148108 parsl.executors.status_handling INFO: Allocated block ID 163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-163 2024-03-17 04:06:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 107 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 107 2024-03-17 04:06:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f744fa553b49aa1d36261e5d5235247 with result from task 1459 parsl.dataflow.dflow DEBUG: Task 1459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d80f9f63-692a-42de-9950-ece2bbfdfb66_isrForCrosstalkSources_1031737_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/d80f9f63-692a-42de-9950-ece2bbfdfb66_isrForCrosstalkSources_1031737_24.stderr 2024-03-17 04:06:21 proc_lsst.multi:170 [INFO] [multi] job_id 17148109 proc_lsst.multi INFO: [multi] job_id 17148109 2024-03-17 04:06:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 108 2024-03-17 04:06:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148109 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148109 parsl.executors.status_handling DEBUG: Launched block 163 on executor multi with job ID 17148109 parsl.executors.status_handling INFO: Allocated block ID 164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-164 2024-03-17 04:06:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 108 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7d8e6a8ded6aa2c3d4ec3038b292e57 with result from task 1617 parsl.dataflow.dflow DEBUG: Task 1617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/fe13a148-4aa4-48b7-acae-fccd94c80e9e_isrForCrosstalkSources_1031706_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/fe13a148-4aa4-48b7-acae-fccd94c80e9e_isrForCrosstalkSources_1031706_41.stderr parsl.dataflow.dflow INFO: Task 1561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b64aa0a5499c34502bb65b7a17fb2c4 with result from task 1561 parsl.dataflow.dflow DEBUG: Task 1561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9b437c32-0194-4bf9-8254-1d6a6ca533e9_isrForCrosstalkSources_1031716_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9b437c32-0194-4bf9-8254-1d6a6ca533e9_isrForCrosstalkSources_1031716_22.stderr 2024-03-17 04:06:21 proc_lsst.multi:170 [INFO] [multi] job_id 17148110 proc_lsst.multi INFO: [multi] job_id 17148110 2024-03-17 04:06:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 109 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 109 2024-03-17 04:06:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148110 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148110 parsl.executors.status_handling DEBUG: Launched block 164 on executor multi with job ID 17148110 parsl.executors.status_handling INFO: Allocated block ID 165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-165 2024-03-17 04:06:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 109 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 109 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a93e6e22e5913d435766f82d4bba9923 with result from task 1657 parsl.dataflow.dflow DEBUG: Task 1657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/22ea7043-0edf-4128-9133-9d4dbc44cbf5_isrForCrosstalkSources_1031737_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/22ea7043-0edf-4128-9133-9d4dbc44cbf5_isrForCrosstalkSources_1031737_61.stderr parsl.dataflow.dflow INFO: Task 1187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b8144e0440558ef3cef4e8cac0cedc3 with result from task 1187 parsl.dataflow.dflow DEBUG: Task 1187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/20dd828f-0f2b-468b-8175-31fdcff93ac5_isrForCrosstalkSources_1031712_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/20dd828f-0f2b-468b-8175-31fdcff93ac5_isrForCrosstalkSources_1031712_57.stderr parsl.dataflow.dflow INFO: Task 1643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0660b155d0547e055168a9c140eb829e with result from task 1643 parsl.dataflow.dflow DEBUG: Task 1643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/fd1bc628-4332-4885-8e88-d8decebfb07d_isrForCrosstalkSources_1031728_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/fd1bc628-4332-4885-8e88-d8decebfb07d_isrForCrosstalkSources_1031728_51.stderr parsl.dataflow.dflow INFO: Task 1594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c34ccb79fc27de31d915cf4aad414c5 with result from task 1594 parsl.dataflow.dflow DEBUG: Task 1594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/a3e91b86-5eba-4c93-97f1-6c56672e6f90_isrForCrosstalkSources_1031700_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/a3e91b86-5eba-4c93-97f1-6c56672e6f90_isrForCrosstalkSources_1031700_46.stderr parsl.dataflow.dflow INFO: Task 1178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd12ca6aa64d99056b8528930996dfa3 with result from task 1178 parsl.dataflow.dflow DEBUG: Task 1178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/cc7ef2ef-47e3-46e2-915c-bbe0e941b7ad_isrForCrosstalkSources_1031718_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/cc7ef2ef-47e3-46e2-915c-bbe0e941b7ad_isrForCrosstalkSources_1031718_62.stderr 2024-03-17 04:06:21 proc_lsst.multi:170 [INFO] [multi] job_id 17148111 proc_lsst.multi INFO: [multi] job_id 17148111 2024-03-17 04:06:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 110 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 110 2024-03-17 04:06:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148111 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148111 parsl.executors.status_handling DEBUG: Launched block 165 on executor multi with job ID 17148111 parsl.executors.status_handling INFO: Allocated block ID 166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-166 2024-03-17 04:06:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 110 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 110 2024-03-17 04:06:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abc5819fd2081d9f6cbb8ff2e364bddf with result from task 1605 parsl.dataflow.dflow DEBUG: Task 1605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/5c986052-af36-47c5-a067-de77cbaef626_isrForCrosstalkSources_1031723_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/5c986052-af36-47c5-a067-de77cbaef626_isrForCrosstalkSources_1031723_39.stderr 2024-03-17 04:06:22 proc_lsst.multi:170 [INFO] [multi] job_id 17148112 proc_lsst.multi INFO: [multi] job_id 17148112 2024-03-17 04:06:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 111 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 111 2024-03-17 04:06:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148112 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148112 parsl.executors.status_handling DEBUG: Launched block 166 on executor multi with job ID 17148112 parsl.executors.status_handling INFO: Allocated block ID 167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-167 2024-03-17 04:06:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 2024-03-17 04:06:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 111 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 111 2024-03-17 04:06:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff2c8a6f8620a04edd01706791a97760 with result from task 1535 parsl.dataflow.dflow DEBUG: Task 1535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/210d18a2-0ffe-447b-824e-31ce5f7fafa2_isrForCrosstalkSources_1031710_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/210d18a2-0ffe-447b-824e-31ce5f7fafa2_isrForCrosstalkSources_1031710_47.stderr parsl.dataflow.dflow INFO: Task 1550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bb039525665d6067196d542bc2bbb14 with result from task 1550 parsl.dataflow.dflow DEBUG: Task 1550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5b8ecf54-f919-4311-a826-0591fea57b07_isrForCrosstalkSources_1031730_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/5b8ecf54-f919-4311-a826-0591fea57b07_isrForCrosstalkSources_1031730_54.stderr parsl.dataflow.dflow INFO: Task 1504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ad67d74eab9c6ceb86020fe5852e2ce with result from task 1504 parsl.dataflow.dflow DEBUG: Task 1504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/43b1614f-d4e1-4362-9674-54c53fe757ee_isrForCrosstalkSources_1031734_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/43b1614f-d4e1-4362-9674-54c53fe757ee_isrForCrosstalkSources_1031734_57.stderr parsl.dataflow.dflow INFO: Task 1584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9936a464135e81f1d99b4b53a2f0200 with result from task 1584 parsl.dataflow.dflow DEBUG: Task 1584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8d38e5c2-fc67-4191-98d9-e55fbbd7277b_isrForCrosstalkSources_1031708_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8d38e5c2-fc67-4191-98d9-e55fbbd7277b_isrForCrosstalkSources_1031708_16.stderr parsl.dataflow.dflow INFO: Task 1412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8862e96f17cd289891744d45b6f7cd6a with result from task 1412 parsl.dataflow.dflow DEBUG: Task 1412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/709f9cf4-5464-478e-b2cc-3e8250189553_isrForCrosstalkSources_1031725_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/709f9cf4-5464-478e-b2cc-3e8250189553_isrForCrosstalkSources_1031725_61.stderr 2024-03-17 04:06:22 proc_lsst.multi:170 [INFO] [multi] job_id 17148113 proc_lsst.multi INFO: [multi] job_id 17148113 2024-03-17 04:06:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 112 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 112 2024-03-17 04:06:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148113 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148113 parsl.executors.status_handling DEBUG: Launched block 167 on executor multi with job ID 17148113 parsl.executors.status_handling INFO: Allocated block ID 168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-168 2024-03-17 04:06:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 2024-03-17 04:06:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 112 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 112 2024-03-17 04:06:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7edb8acff471c25da91d06fd43ef002 with result from task 1554 parsl.dataflow.dflow DEBUG: Task 1554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/456cfab5-7933-4d09-ba0f-fa9692656b62_isrForCrosstalkSources_1031714_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/456cfab5-7933-4d09-ba0f-fa9692656b62_isrForCrosstalkSources_1031714_53.stderr parsl.dataflow.dflow INFO: Task 1182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75240c42b9b63f43634098fdab08d8ac with result from task 1182 parsl.dataflow.dflow DEBUG: Task 1182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/22a6ee97-64c2-43dd-911b-d9058602700e_isrForCrosstalkSources_1031700_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/22a6ee97-64c2-43dd-911b-d9058602700e_isrForCrosstalkSources_1031700_57.stderr parsl.dataflow.dflow INFO: Task 1546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa1837101d2358e5a44d4b968c63c72b with result from task 1546 parsl.dataflow.dflow DEBUG: Task 1546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e92d6e1d-6cd3-48ab-a815-59083a3d4e0e_isrForCrosstalkSources_1031712_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e92d6e1d-6cd3-48ab-a815-59083a3d4e0e_isrForCrosstalkSources_1031712_54.stderr parsl.dataflow.dflow INFO: Task 1573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d202ab83e41093cc0e4ef36f8abc080 with result from task 1573 parsl.dataflow.dflow DEBUG: Task 1573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fca500ee-19e6-4f85-999f-16ad3304f797_isrForCrosstalkSources_1031707_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fca500ee-19e6-4f85-999f-16ad3304f797_isrForCrosstalkSources_1031707_58.stderr parsl.dataflow.dflow INFO: Task 1553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d2e00ffb3627820bdc50b25bbdcdced with result from task 1553 parsl.dataflow.dflow DEBUG: Task 1553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/fb5bf47c-b970-4f8a-bbae-cbbf15f7172b_isrForCrosstalkSources_1031728_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/fb5bf47c-b970-4f8a-bbae-cbbf15f7172b_isrForCrosstalkSources_1031728_59.stderr 2024-03-17 04:06:23 proc_lsst.multi:170 [INFO] [multi] job_id 17148114 proc_lsst.multi INFO: [multi] job_id 17148114 2024-03-17 04:06:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 113 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 113 2024-03-17 04:06:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148114 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148114 parsl.executors.status_handling DEBUG: Launched block 168 on executor multi with job ID 17148114 parsl.executors.status_handling INFO: Allocated block ID 169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-169 2024-03-17 04:06:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 parsl.dataflow.dflow INFO: Task 1361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b85ac5e6d5c14ff21c64c0e886ca00a3 with result from task 1361 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 113 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 113 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.dflow DEBUG: Task 1361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/9d8eae95-8e56-4009-b154-f8f448f650ea_isrForCrosstalkSources_1031737_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/9d8eae95-8e56-4009-b154-f8f448f650ea_isrForCrosstalkSources_1031737_34.stderr proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f05bae2e3da50ca94b792907852320a with result from task 1608 parsl.dataflow.dflow DEBUG: Task 1608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/05468c1e-1b84-4f19-b55d-83a1a493dba7_isrForCrosstalkSources_1031701_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/05468c1e-1b84-4f19-b55d-83a1a493dba7_isrForCrosstalkSources_1031701_24.stderr parsl.dataflow.dflow INFO: Task 1639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76e8e794e817f1565307b29f64c834ae with result from task 1639 parsl.dataflow.dflow DEBUG: Task 1639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ef38f34d-c398-412b-8839-f1ca76886b49_isrForCrosstalkSources_1031699_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ef38f34d-c398-412b-8839-f1ca76886b49_isrForCrosstalkSources_1031699_1.stderr parsl.dataflow.dflow INFO: Task 1180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca41c86d291b31ee6c4e3fd948e7c194 with result from task 1180 parsl.dataflow.dflow DEBUG: Task 1180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4c5d2dd1-3eb2-468c-891c-6f4de05d7a82_isrForCrosstalkSources_1031719_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4c5d2dd1-3eb2-468c-891c-6f4de05d7a82_isrForCrosstalkSources_1031719_52.stderr parsl.dataflow.dflow INFO: Task 1634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13ee6ff3975ab4f4a53a09db9f144405 with result from task 1634 parsl.dataflow.dflow DEBUG: Task 1634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8b7cb6a0-23f9-4aca-9ccd-06caad33be52_isrForCrosstalkSources_1031729_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8b7cb6a0-23f9-4aca-9ccd-06caad33be52_isrForCrosstalkSources_1031729_28.stderr 2024-03-17 04:06:23 proc_lsst.multi:170 [INFO] [multi] job_id 17148115 proc_lsst.multi INFO: [multi] job_id 17148115 2024-03-17 04:06:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 114 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 114 2024-03-17 04:06:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148115 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148115 parsl.executors.status_handling DEBUG: Launched block 169 on executor multi with job ID 17148115 parsl.executors.status_handling INFO: Allocated block ID 170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-170 2024-03-17 04:06:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 114 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 114 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c7641d8a3103d96959726104ed169cb with result from task 1895 parsl.dataflow.dflow DEBUG: Task 1895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/3bfce2e5-bd57-4867-8002-d89e4252b536_isrForCrosstalkSources_1031726_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/3bfce2e5-bd57-4867-8002-d89e4252b536_isrForCrosstalkSources_1031726_25.stderr 2024-03-17 04:06:23 proc_lsst.multi:170 [INFO] [multi] job_id 17148116 proc_lsst.multi INFO: [multi] job_id 17148116 2024-03-17 04:06:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 115 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 115 2024-03-17 04:06:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148116 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148116 parsl.executors.status_handling DEBUG: Launched block 170 on executor multi with job ID 17148116 parsl.executors.status_handling INFO: Allocated block ID 171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-171 2024-03-17 04:06:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 115 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 115 2024-03-17 04:06:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 666837833b3cbd70b0756c1cddfd0c3f with result from task 1510 parsl.dataflow.dflow DEBUG: Task 1510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/11d7635c-079b-4f5e-8b9f-75f7fb5bb838_isrForCrosstalkSources_1031706_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/11d7635c-079b-4f5e-8b9f-75f7fb5bb838_isrForCrosstalkSources_1031706_23.stderr parsl.dataflow.dflow INFO: Task 1527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 928db3e15fff594e516cd3e6a5ab371e with result from task 1527 parsl.dataflow.dflow DEBUG: Task 1527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/013681b6-b80a-49df-b6e5-a9aae18a846f_isrForCrosstalkSources_1031720_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/013681b6-b80a-49df-b6e5-a9aae18a846f_isrForCrosstalkSources_1031720_43.stderr parsl.dataflow.dflow INFO: Task 1598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee33ace8f871a798d17b3a2647292980 with result from task 1598 parsl.dataflow.dflow DEBUG: Task 1598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/c434acb4-dee3-43cb-bb06-5f8d449255cc_isrForCrosstalkSources_1031717_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/c434acb4-dee3-43cb-bb06-5f8d449255cc_isrForCrosstalkSources_1031717_47.stderr parsl.dataflow.dflow INFO: Task 1360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 185e8b9f0368595a5d2b857205c02d7f with result from task 1360 parsl.dataflow.dflow DEBUG: Task 1360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/2c5f1532-e2cb-4cd3-ae83-4f30d2c94ab2_isrForCrosstalkSources_1031714_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/2c5f1532-e2cb-4cd3-ae83-4f30d2c94ab2_isrForCrosstalkSources_1031714_32.stderr parsl.dataflow.dflow INFO: Task 1599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0ca308e230add832084c224663c9053 with result from task 1599 parsl.dataflow.dflow DEBUG: Task 1599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f7683daf-4483-4b0e-bcb5-4a28605a5bac_isrForCrosstalkSources_1031703_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f7683daf-4483-4b0e-bcb5-4a28605a5bac_isrForCrosstalkSources_1031703_62.stderr parsl.dataflow.dflow INFO: Task 1384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c54510b51031d60fec28ed706d1d32f3 with result from task 1384 parsl.dataflow.dflow DEBUG: Task 1384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/e5a4ecb9-3a8e-4684-ad38-7aa89a00dd45_isrForCrosstalkSources_1031705_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/e5a4ecb9-3a8e-4684-ad38-7aa89a00dd45_isrForCrosstalkSources_1031705_4.stderr parsl.dataflow.dflow INFO: Task 1315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54df400acdbc4b438b2c8b02a14b530f with result from task 1315 parsl.dataflow.dflow DEBUG: Task 1315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7a4d1660-9fed-49fa-9c01-537f2cb600be_isrForCrosstalkSources_1031701_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7a4d1660-9fed-49fa-9c01-537f2cb600be_isrForCrosstalkSources_1031701_16.stderr 2024-03-17 04:06:24 proc_lsst.multi:170 [INFO] [multi] job_id 17148117 proc_lsst.multi INFO: [multi] job_id 17148117 2024-03-17 04:06:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 116 2024-03-17 04:06:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148117 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148117 parsl.executors.status_handling DEBUG: Launched block 171 on executor multi with job ID 17148117 parsl.executors.status_handling INFO: Allocated block ID 172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-172 2024-03-17 04:06:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 2024-03-17 04:06:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 116 2024-03-17 04:06:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3ef7ec35434a9558637d923d7e0af98 with result from task 1382 parsl.dataflow.dflow DEBUG: Task 1382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/77924852-fc69-4f0e-bb07-12742aa97908_isrForCrosstalkSources_1031715_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/77924852-fc69-4f0e-bb07-12742aa97908_isrForCrosstalkSources_1031715_19.stderr parsl.dataflow.dflow INFO: Task 1248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d14306c92177df7e437bd640f7c5530f with result from task 1248 parsl.dataflow.dflow DEBUG: Task 1248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/3eaf4001-1db2-4856-9e98-0b2799d7cd18_isrForCrosstalkSources_1031702_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/3eaf4001-1db2-4856-9e98-0b2799d7cd18_isrForCrosstalkSources_1031702_1.stderr parsl.dataflow.dflow INFO: Task 1549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bf1c796f5e0aa13a525e72d255c9b37 with result from task 1549 parsl.dataflow.dflow DEBUG: Task 1549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/598ec560-e8d4-4c59-b7cf-7d05900830c2_isrForCrosstalkSources_1031733_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/598ec560-e8d4-4c59-b7cf-7d05900830c2_isrForCrosstalkSources_1031733_54.stderr parsl.dataflow.dflow INFO: Task 1497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c501af1e6eb198e984462797c000ed with result from task 1497 parsl.dataflow.dflow DEBUG: Task 1497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0bc8f33e-d523-46a4-a870-576e5c603ddb_isrForCrosstalkSources_1031711_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/0bc8f33e-d523-46a4-a870-576e5c603ddb_isrForCrosstalkSources_1031711_54.stderr parsl.dataflow.dflow INFO: Task 1590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f29f0032001a803668cb3b7c7c33cd05 with result from task 1590 parsl.dataflow.dflow DEBUG: Task 1590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/65da1710-8b12-49de-aa6b-d5193cd4c536_isrForCrosstalkSources_1031715_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/65da1710-8b12-49de-aa6b-d5193cd4c536_isrForCrosstalkSources_1031715_61.stderr parsl.dataflow.dflow INFO: Task 1322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad56c459e10d10e308fad83caa548bfd with result from task 1322 parsl.dataflow.dflow DEBUG: Task 1322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/b4db0d1f-e53c-4430-81ad-a4bf916f4f1e_isrForCrosstalkSources_1031720_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/b4db0d1f-e53c-4430-81ad-a4bf916f4f1e_isrForCrosstalkSources_1031720_18.stderr parsl.dataflow.dflow INFO: Task 1179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15775b2de295c42338c7a07a82c99904 with result from task 1179 parsl.dataflow.dflow DEBUG: Task 1179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks 2024-03-17 04:06:24 proc_lsst.multi:170 [INFO] [multi] job_id 17148118 proc_lsst.multi INFO: [multi] job_id 17148118 2024-03-17 04:06:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 117 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 117 2024-03-17 04:06:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148118 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148118 parsl.dataflow.dflow INFO: Standard output for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c853fdf4-9e13-43c2-a77e-65fca2dc50b0_isrForCrosstalkSources_1031732_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c853fdf4-9e13-43c2-a77e-65fca2dc50b0_isrForCrosstalkSources_1031732_59.stderr parsl.executors.status_handling DEBUG: Launched block 172 on executor multi with job ID 17148118 parsl.executors.status_handling INFO: Allocated block ID 173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-173 2024-03-17 04:06:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 2024-03-17 04:06:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 117 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 117 2024-03-17 04:06:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99abb1d08fa82b15a0c5252e3aaf250a with result from task 1877 parsl.dataflow.dflow DEBUG: Task 1877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b7e58c0c-3440-4806-8757-c0a29ad377d8_isrForCrosstalkSources_1031728_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b7e58c0c-3440-4806-8757-c0a29ad377d8_isrForCrosstalkSources_1031728_23.stderr parsl.dataflow.dflow INFO: Task 1481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27320fabff1fa44226469de5c29750aa with result from task 1481 parsl.dataflow.dflow DEBUG: Task 1481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f163cb32-de9e-425a-bc60-9eead08bbfc9_isrForCrosstalkSources_1031737_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f163cb32-de9e-425a-bc60-9eead08bbfc9_isrForCrosstalkSources_1031737_18.stderr parsl.dataflow.dflow INFO: Task 1386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 813e2d4ab555377d3e0a572cbeb31379 with result from task 1386 parsl.dataflow.dflow DEBUG: Task 1386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/32ec9f6a-aff7-4500-aefa-790624ea0996_isrForCrosstalkSources_1031705_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/32ec9f6a-aff7-4500-aefa-790624ea0996_isrForCrosstalkSources_1031705_20.stderr parsl.dataflow.dflow INFO: Task 1492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00e3c623046ef223029ec98534fc2533 with result from task 1492 parsl.dataflow.dflow DEBUG: Task 1492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/6927d862-5195-4b98-ab22-2f5ca2b1d61c_isrForCrosstalkSources_1031727_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/6927d862-5195-4b98-ab22-2f5ca2b1d61c_isrForCrosstalkSources_1031727_11.stderr parsl.dataflow.dflow INFO: Task 1604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 139fed699ba95c5387411d29b927cd0c with result from task 1604 parsl.dataflow.dflow DEBUG: Task 1604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/165ccdc3-efa9-4e6d-8546-c337fcaee0f0_isrForCrosstalkSources_1031702_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/165ccdc3-efa9-4e6d-8546-c337fcaee0f0_isrForCrosstalkSources_1031702_16.stderr 2024-03-17 04:06:25 proc_lsst.multi:170 [INFO] [multi] job_id 17148119 proc_lsst.multi INFO: [multi] job_id 17148119 2024-03-17 04:06:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 118 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 118 2024-03-17 04:06:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148119 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148119 parsl.executors.status_handling DEBUG: Launched block 173 on executor multi with job ID 17148119 parsl.executors.status_handling INFO: Allocated block ID 174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-174 2024-03-17 04:06:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 118 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 118 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a751a4d3cd3741ddf5fbe49078ba9efd with result from task 1899 parsl.dataflow.dflow DEBUG: Task 1899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7d547375-1f31-49f6-923d-4af9699848b1_isrForCrosstalkSources_1031702_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7d547375-1f31-49f6-923d-4af9699848b1_isrForCrosstalkSources_1031702_20.stderr parsl.dataflow.dflow INFO: Task 1314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 090868f1dd5d12baf3e5daaed25e60e6 with result from task 1314 parsl.dataflow.dflow DEBUG: Task 1314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/666ae8a8-329b-40fc-9d50-08e365aae8b0_isrForCrosstalkSources_1031721_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/666ae8a8-329b-40fc-9d50-08e365aae8b0_isrForCrosstalkSources_1031721_15.stderr parsl.dataflow.dflow INFO: Task 1316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be1da325039898b1082392f4fb2382db with result from task 1316 parsl.dataflow.dflow DEBUG: Task 1316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/31f76ab4-b32e-4c90-b717-48d58d842c24_isrForCrosstalkSources_1031701_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/31f76ab4-b32e-4c90-b717-48d58d842c24_isrForCrosstalkSources_1031701_42.stderr parsl.dataflow.dflow INFO: Task 1478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 124974eff54050b2427d18b5fd09612f with result from task 1478 parsl.dataflow.dflow DEBUG: Task 1478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/07141926-6f00-484c-ba4c-b6d33dddf06d_isrForCrosstalkSources_1031717_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/07141926-6f00-484c-ba4c-b6d33dddf06d_isrForCrosstalkSources_1031717_61.stderr 2024-03-17 04:06:25 proc_lsst.multi:170 [INFO] [multi] job_id 17148120 proc_lsst.multi INFO: [multi] job_id 17148120 2024-03-17 04:06:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 119 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 119 2024-03-17 04:06:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148120 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148120 parsl.executors.status_handling DEBUG: Launched block 174 on executor multi with job ID 17148120 parsl.executors.status_handling INFO: Allocated block ID 175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-175 2024-03-17 04:06:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 119 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 119 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23525a964641b65cbd205fe7c0f41423 with result from task 1610 parsl.dataflow.dflow DEBUG: Task 1610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/01503cc5-1c10-4c1f-874c-69c265571382_isrForCrosstalkSources_1031707_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/01503cc5-1c10-4c1f-874c-69c265571382_isrForCrosstalkSources_1031707_27.stderr parsl.dataflow.dflow INFO: Task 1621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 639b957b92e2a12e3adc9745b48d56bd with result from task 1621 parsl.dataflow.dflow DEBUG: Task 1621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8c1a3e0a-95bc-4520-8547-a0df5474eb8a_isrForCrosstalkSources_1031729_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8c1a3e0a-95bc-4520-8547-a0df5474eb8a_isrForCrosstalkSources_1031729_25.stderr parsl.dataflow.dflow INFO: Task 1317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1103236a98fe2448eb2e6335957d2f44 with result from task 1317 parsl.dataflow.dflow DEBUG: Task 1317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/2f76929d-af22-42da-852d-e55341c159da_isrForCrosstalkSources_1031701_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/2f76929d-af22-42da-852d-e55341c159da_isrForCrosstalkSources_1031701_50.stderr parsl.dataflow.dflow INFO: Task 1659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10797f4609760cf764b625ba5356056a with result from task 1659 parsl.dataflow.dflow DEBUG: Task 1659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/015cdbab-0c64-41ac-a4c8-fbca861347e8_isrForCrosstalkSources_1031702_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/015cdbab-0c64-41ac-a4c8-fbca861347e8_isrForCrosstalkSources_1031702_26.stderr parsl.dataflow.dflow INFO: Task 1901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3eff06dc89823cd5d9c25d92b87e5faa with result from task 1901 parsl.dataflow.dflow DEBUG: Task 1901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b336c411-a3de-4412-802d-0bd081bbb7c8_isrForCrosstalkSources_1031716_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b336c411-a3de-4412-802d-0bd081bbb7c8_isrForCrosstalkSources_1031716_45.stderr 2024-03-17 04:06:25 proc_lsst.multi:170 [INFO] [multi] job_id 17148121 proc_lsst.multi INFO: [multi] job_id 17148121 2024-03-17 04:06:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 120 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 120 2024-03-17 04:06:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148121 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148121 parsl.executors.status_handling DEBUG: Launched block 175 on executor multi with job ID 17148121 parsl.executors.status_handling INFO: Allocated block ID 176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-176 2024-03-17 04:06:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 120 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 120 2024-03-17 04:06:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89e28b30075bb2a54221c86c76397e13 with result from task 1622 parsl.dataflow.dflow DEBUG: Task 1622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8ca7793c-534c-47d7-9cb0-4226fb4575d2_isrForCrosstalkSources_1031707_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8ca7793c-534c-47d7-9cb0-4226fb4575d2_isrForCrosstalkSources_1031707_26.stderr parsl.dataflow.dflow INFO: Task 1660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5d3591a09ded86b152c19dc69f8df8c with result from task 1660 parsl.dataflow.dflow DEBUG: Task 1660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a35ee0ad-c351-41c9-86c9-075c4852a6db_isrForCrosstalkSources_1031712_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a35ee0ad-c351-41c9-86c9-075c4852a6db_isrForCrosstalkSources_1031712_26.stderr parsl.dataflow.dflow INFO: Task 1495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 179cc75748892a296c093c183d496828 with result from task 1495 parsl.dataflow.dflow DEBUG: Task 1495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/535e482a-d707-401c-a5b3-81c53c42f204_isrForCrosstalkSources_1031714_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/535e482a-d707-401c-a5b3-81c53c42f204_isrForCrosstalkSources_1031714_5.stderr parsl.dataflow.dflow INFO: Task 1652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d080879110d2379de47422fa12b7cf6 with result from task 1652 parsl.dataflow.dflow DEBUG: Task 1652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/3fd6a7f5-667a-4479-848c-2ed51b99b147_isrForCrosstalkSources_1031724_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/3fd6a7f5-667a-4479-848c-2ed51b99b147_isrForCrosstalkSources_1031724_36.stderr parsl.dataflow.dflow INFO: Task 1612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adf9e1f7ffda09b83901dba8e3b55e1b with result from task 1612 parsl.dataflow.dflow DEBUG: Task 1612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/efcd86b3-10bf-4be6-a28f-6d97871af519_isrForCrosstalkSources_1031699_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/efcd86b3-10bf-4be6-a28f-6d97871af519_isrForCrosstalkSources_1031699_38.stderr 2024-03-17 04:06:26 proc_lsst.multi:170 [INFO] [multi] job_id 17148122 proc_lsst.multi INFO: [multi] job_id 17148122 2024-03-17 04:06:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 121 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 121 2024-03-17 04:06:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148122 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148122 parsl.executors.status_handling DEBUG: Launched block 176 on executor multi with job ID 17148122 parsl.executors.status_handling INFO: Allocated block ID 177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-177 2024-03-17 04:06:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 2024-03-17 04:06:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 121 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 121 2024-03-17 04:06:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65cc9cca37515ee5df0c4e12f2289e58 with result from task 1661 parsl.dataflow.dflow DEBUG: Task 1661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0c69dd18-5fe2-43c8-867f-f1afe07f770d_isrForCrosstalkSources_1031721_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0c69dd18-5fe2-43c8-867f-f1afe07f770d_isrForCrosstalkSources_1031721_62.stderr parsl.dataflow.dflow INFO: Task 1487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5557e74ce1089d3c1f8df26726fc8931 with result from task 1487 parsl.dataflow.dflow DEBUG: Task 1487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e63d0527-2812-4652-8aeb-9b6580c86023_isrForCrosstalkSources_1031736_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e63d0527-2812-4652-8aeb-9b6580c86023_isrForCrosstalkSources_1031736_29.stderr parsl.dataflow.dflow INFO: Task 1648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2205e5abcb07edee57baa883f3ed6623 with result from task 1648 parsl.dataflow.dflow DEBUG: Task 1648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/01cf5685-80a0-4a1f-9b0c-f425c875c404_isrForCrosstalkSources_1031718_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/01cf5685-80a0-4a1f-9b0c-f425c875c404_isrForCrosstalkSources_1031718_49.stderr parsl.dataflow.dflow INFO: Task 1658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a9d8e101dcc74d951952e44986a355c with result from task 1658 parsl.dataflow.dflow DEBUG: Task 1658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/f0b7e38f-171e-4201-98d3-f9e9958004fd_isrForCrosstalkSources_1031736_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/f0b7e38f-171e-4201-98d3-f9e9958004fd_isrForCrosstalkSources_1031736_38.stderr parsl.dataflow.dflow INFO: Task 1812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcf94961d9372337ccb077263d0c3408 with result from task 1812 parsl.dataflow.dflow DEBUG: Task 1812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/de22fed1-f3c3-4b3f-82cc-4d82ce123d25_isrForCrosstalkSources_1031717_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/de22fed1-f3c3-4b3f-82cc-4d82ce123d25_isrForCrosstalkSources_1031717_13.stderr 2024-03-17 04:06:27 proc_lsst.multi:170 [INFO] [multi] job_id 17148123 proc_lsst.multi INFO: [multi] job_id 17148123 2024-03-17 04:06:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 122 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 122 2024-03-17 04:06:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148123 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148123 parsl.executors.status_handling DEBUG: Launched block 177 on executor multi with job ID 17148123 parsl.executors.status_handling INFO: Allocated block ID 178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-178 2024-03-17 04:06:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 2024-03-17 04:06:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 122 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 122 2024-03-17 04:06:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fbe689984e75cc983e2edb62d2d5ff3 with result from task 1646 parsl.dataflow.dflow DEBUG: Task 1646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/be370e8d-aae8-4faf-84b7-3e69539d374d_isrForCrosstalkSources_1031708_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/be370e8d-aae8-4faf-84b7-3e69539d374d_isrForCrosstalkSources_1031708_27.stderr parsl.dataflow.dflow INFO: Task 1580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e71af292b006a27dade0b426dda5c44 with result from task 1580 parsl.dataflow.dflow DEBUG: Task 1580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/ecdab980-80bf-4bcb-9b0d-0386887b7581_isrForCrosstalkSources_1031732_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/ecdab980-80bf-4bcb-9b0d-0386887b7581_isrForCrosstalkSources_1031732_6.stderr parsl.dataflow.dflow INFO: Task 1642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8583de7e9a17ef6000f8aaa5cb2e96be with result from task 1642 parsl.dataflow.dflow DEBUG: Task 1642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/5a1b896f-9c0c-417b-a6a3-cdbf31a248f5_isrForCrosstalkSources_1031707_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/5a1b896f-9c0c-417b-a6a3-cdbf31a248f5_isrForCrosstalkSources_1031707_39.stderr parsl.dataflow.dflow INFO: Task 1662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c60d3871011926cc741d7bc2c724f954 with result from task 1662 parsl.dataflow.dflow DEBUG: Task 1662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/da0a194f-c964-4ef7-9c08-1072da913d63_isrForCrosstalkSources_1031734_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/da0a194f-c964-4ef7-9c08-1072da913d63_isrForCrosstalkSources_1031734_8.stderr parsl.dataflow.dflow INFO: Task 1798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a0dace2bd07ee770f7723c6370ff211 with result from task 1798 parsl.dataflow.dflow DEBUG: Task 1798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6078dcfd-3465-4eb2-b518-38ab70b7e178_isrForCrosstalkSources_1031726_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6078dcfd-3465-4eb2-b518-38ab70b7e178_isrForCrosstalkSources_1031726_32.stderr parsl.dataflow.dflow INFO: Task 1898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 495735df98930a591b771373f3cca421 with result from task 1898 parsl.dataflow.dflow DEBUG: Task 1898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c7ae58d5-acb1-4ca2-bf20-e3226593333d_isrForCrosstalkSources_1031728_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c7ae58d5-acb1-4ca2-bf20-e3226593333d_isrForCrosstalkSources_1031728_55.stderr parsl.dataflow.dflow INFO: Task 1258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d8021f54efc6ae9e8059b33675e611b with result from task 1258 parsl.dataflow.dflow DEBUG: Task 1258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/c2849103-1256-40c2-837b-fa1d779729c1_isrForCrosstalkSources_1031713_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/c2849103-1256-40c2-837b-fa1d779729c1_isrForCrosstalkSources_1031713_38.stderr parsl.dataflow.dflow INFO: Task 1252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d052f5068e06922c0c36904a49624b32 with result from task 1252 parsl.dataflow.dflow DEBUG: Task 1252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/dc046bde-3677-4281-aba1-9043a067dbaf_isrForCrosstalkSources_1031709_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/dc046bde-3677-4281-aba1-9043a067dbaf_isrForCrosstalkSources_1031709_35.stderr parsl.dataflow.dflow INFO: Task 1908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c8fdbc56f3c1df0cc8f2c5b499665c0 with result from task 1908 parsl.dataflow.dflow DEBUG: Task 1908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/1eb7b5ad-c1d4-4211-9d30-fb0422cfa807_isrForCrosstalkSources_1031730_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/1eb7b5ad-c1d4-4211-9d30-fb0422cfa807_isrForCrosstalkSources_1031730_12.stderr parsl.dataflow.dflow INFO: Task 1775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 624d2109f6f341db11b90b47c7b5c67b with result from task 1775 parsl.dataflow.dflow DEBUG: Task 1775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/5d296789-3732-4331-af60-da530473fd21_isrForCrosstalkSources_1031737_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/5d296789-3732-4331-af60-da530473fd21_isrForCrosstalkSources_1031737_25.stderr 2024-03-17 04:06:27 proc_lsst.multi:170 [INFO] [multi] job_id 17148124 proc_lsst.multi INFO: [multi] job_id 17148124 2024-03-17 04:06:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 123 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 123 2024-03-17 04:06:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148124 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148124 parsl.executors.status_handling DEBUG: Launched block 178 on executor multi with job ID 17148124 parsl.executors.status_handling INFO: Allocated block ID 179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-179 2024-03-17 04:06:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 2024-03-17 04:06:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 123 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 123 2024-03-17 04:06:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f1040d6cafcc1e14f7f8bf29bf7e293 with result from task 1257 parsl.dataflow.dflow DEBUG: Task 1257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/708de2a3-9ed6-407a-b5cd-ccfa0e6dcc12_isrForCrosstalkSources_1031709_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/708de2a3-9ed6-407a-b5cd-ccfa0e6dcc12_isrForCrosstalkSources_1031709_28.stderr parsl.dataflow.dflow INFO: Task 1907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3eb9692f5e7626a514eb2288e5416399 with result from task 1907 parsl.dataflow.dflow DEBUG: Task 1907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9fa72e42-7d99-4b60-afad-ca4743303d64_isrForCrosstalkSources_1031718_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9fa72e42-7d99-4b60-afad-ca4743303d64_isrForCrosstalkSources_1031718_27.stderr parsl.dataflow.dflow INFO: Task 1859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d385a38240fe28d6e25dfd467127f753 with result from task 1859 parsl.dataflow.dflow DEBUG: Task 1859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ca347d49-567a-4be6-b6c4-3f51c933388a_isrForCrosstalkSources_1031738_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/ca347d49-567a-4be6-b6c4-3f51c933388a_isrForCrosstalkSources_1031738_34.stderr parsl.dataflow.dflow INFO: Task 1906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81896b0d7ab4644ea6609e0d757a8a27 with result from task 1906 parsl.dataflow.dflow DEBUG: Task 1906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bbfb01d1-c7da-4900-a2b8-9ee06209505b_isrForCrosstalkSources_1031729_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/bbfb01d1-c7da-4900-a2b8-9ee06209505b_isrForCrosstalkSources_1031729_33.stderr parsl.dataflow.dflow INFO: Task 1568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 786947cba9e3ead8dd17cff9ac9cf904 with result from task 1568 parsl.dataflow.dflow DEBUG: Task 1568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a117b22f-ee31-41a4-ae8d-5532dd6f46a0_isrForCrosstalkSources_1031731_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a117b22f-ee31-41a4-ae8d-5532dd6f46a0_isrForCrosstalkSources_1031731_12.stderr parsl.dataflow.dflow INFO: Task 1651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b65513f7a9d2725ab6821281beffafd with result from task 1651 parsl.dataflow.dflow DEBUG: Task 1651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b6b99375-8d37-4f94-936e-b25424d49838_isrForCrosstalkSources_1031716_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b6b99375-8d37-4f94-936e-b25424d49838_isrForCrosstalkSources_1031716_44.stderr parsl.dataflow.dflow INFO: Task 1490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c32301e9f3c581e0e41768912f4869a3 with result from task 1490 parsl.dataflow.dflow DEBUG: Task 1490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/f48c62f8-66fb-4507-94be-6f09a5a44a80_isrForCrosstalkSources_1031724_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/f48c62f8-66fb-4507-94be-6f09a5a44a80_isrForCrosstalkSources_1031724_52.stderr parsl.dataflow.dflow INFO: Task 1583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c38062165b2d717ad80390f1b2eee03a with result from task 1583 parsl.dataflow.dflow DEBUG: Task 1583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/139b6b1f-6197-49f1-9ec5-1b8a803ea48e_isrForCrosstalkSources_1031732_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/139b6b1f-6197-49f1-9ec5-1b8a803ea48e_isrForCrosstalkSources_1031732_29.stderr parsl.dataflow.dflow INFO: Task 1625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc5e93c9e9cae12faec3dce910d8ad1c with result from task 1625 parsl.dataflow.dflow DEBUG: Task 1625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/2352a158-8fbd-485d-8d66-3e5752e851c0_isrForCrosstalkSources_1031720_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/2352a158-8fbd-485d-8d66-3e5752e851c0_isrForCrosstalkSources_1031720_59.stderr parsl.dataflow.dflow INFO: Task 1619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0baba2a7801cb0eeaf00bb8daa315213 with result from task 1619 parsl.dataflow.dflow DEBUG: Task 1619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c6959664-2b7e-476b-841a-da6f75a3b910_isrForCrosstalkSources_1031738_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/c6959664-2b7e-476b-841a-da6f75a3b910_isrForCrosstalkSources_1031738_53.stderr parsl.dataflow.dflow INFO: Task 1665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8f7fd177c3036c25302cb587f69d125 with result from task 1665 parsl.dataflow.dflow DEBUG: Task 1665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/196fe2ae-762c-4158-bd7e-e076ce1d8a90_isrForCrosstalkSources_1031720_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/196fe2ae-762c-4158-bd7e-e076ce1d8a90_isrForCrosstalkSources_1031720_47.stderr 2024-03-17 04:06:28 proc_lsst.multi:170 [INFO] [multi] job_id 17148125 proc_lsst.multi INFO: [multi] job_id 17148125 2024-03-17 04:06:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 124 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 124 2024-03-17 04:06:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148125 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148125 parsl.executors.status_handling DEBUG: Launched block 179 on executor multi with job ID 17148125 parsl.executors.status_handling INFO: Allocated block ID 180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-180 2024-03-17 04:06:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 2024-03-17 04:06:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 124 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 124 2024-03-17 04:06:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf0c9f291bdf6b7d8f2f6a7a60d13bc6 with result from task 1931 parsl.dataflow.dflow DEBUG: Task 1931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/98a4861f-b62c-4d0e-9273-9b4b8ab09069_isrForCrosstalkSources_1031732_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/98a4861f-b62c-4d0e-9273-9b4b8ab09069_isrForCrosstalkSources_1031732_13.stderr parsl.dataflow.dflow INFO: Task 1934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4ae2e9a0ed95b8421f312fa1838f0b3 with result from task 1934 parsl.dataflow.dflow DEBUG: Task 1934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/446f5cc7-99d2-4a2f-bdab-48815d7d677a_isrForCrosstalkSources_1031725_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/446f5cc7-99d2-4a2f-bdab-48815d7d677a_isrForCrosstalkSources_1031725_8.stderr parsl.dataflow.dflow INFO: Task 1366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e9f5b8aa88c96d7fb1a686bce28153b with result from task 1366 parsl.dataflow.dflow DEBUG: Task 1366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/38a17997-35f4-4e77-826b-8448d7961da2_isrForCrosstalkSources_1031711_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/38a17997-35f4-4e77-826b-8448d7961da2_isrForCrosstalkSources_1031711_10.stderr parsl.dataflow.dflow INFO: Task 1620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e12a96c9bf0495f1d9309d41c8a40d7f with result from task 1620 parsl.dataflow.dflow DEBUG: Task 1620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/9cb60c84-1494-49eb-99cf-99df2793052c_isrForCrosstalkSources_1031728_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/9cb60c84-1494-49eb-99cf-99df2793052c_isrForCrosstalkSources_1031728_8.stderr 2024-03-17 04:06:28 proc_lsst.multi:170 [INFO] [multi] job_id 17148126 proc_lsst.multi INFO: [multi] job_id 17148126 2024-03-17 04:06:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 125 2024-03-17 04:06:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148126 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148126 parsl.executors.status_handling DEBUG: Launched block 180 on executor multi with job ID 17148126 parsl.executors.status_handling INFO: Allocated block ID 181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-181 2024-03-17 04:06:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 2024-03-17 04:06:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 125 2024-03-17 04:06:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b8208f5596804579d0a917feac3bd18 with result from task 1473 parsl.dataflow.dflow DEBUG: Task 1473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0bdba1ed-444c-4a39-b720-b033701e80f8_isrForCrosstalkSources_1031732_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0bdba1ed-444c-4a39-b720-b033701e80f8_isrForCrosstalkSources_1031732_42.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dda462a4a61fbf21b3d06d8f1bb4d810 with result from task 1450 parsl.dataflow.dflow DEBUG: Task 1450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7f9f1b5b-d03c-4231-ab97-9243913ce0bf_isrForCrosstalkSources_1031729_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7f9f1b5b-d03c-4231-ab97-9243913ce0bf_isrForCrosstalkSources_1031729_3.stderr parsl.dataflow.dflow INFO: Task 1653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f3f260be947ac9bc75e86084d0d6a6 with result from task 1653 parsl.dataflow.dflow DEBUG: Task 1653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/8d434c01-3304-4c89-a974-052fedc8d0c0_isrForCrosstalkSources_1031722_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/8d434c01-3304-4c89-a974-052fedc8d0c0_isrForCrosstalkSources_1031722_20.stderr parsl.dataflow.dflow INFO: Task 1936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8333eb468ac879f5589f87fa76bd411 with result from task 1936 parsl.dataflow.dflow DEBUG: Task 1936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/ec38687e-611e-4312-96a2-bf110d25c044_isrForCrosstalkSources_1031719_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/ec38687e-611e-4312-96a2-bf110d25c044_isrForCrosstalkSources_1031719_9.stderr parsl.dataflow.dflow INFO: Task 1385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 205b80225594797a681b73a9a4bd33cc with result from task 1385 parsl.dataflow.dflow DEBUG: Task 1385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5f6ce825-0e86-4e6a-b702-dabc7e0fbe3a_isrForCrosstalkSources_1031711_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/5f6ce825-0e86-4e6a-b702-dabc7e0fbe3a_isrForCrosstalkSources_1031711_17.stderr parsl.dataflow.dflow INFO: Task 1927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b48fe45aa984114abe38dc393942075 with result from task 1927 parsl.dataflow.dflow DEBUG: Task 1927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e6ae05d3-17a8-4e2d-a07e-25f4d586274f_isrForCrosstalkSources_1031731_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/e6ae05d3-17a8-4e2d-a07e-25f4d586274f_isrForCrosstalkSources_1031731_48.stderr parsl.dataflow.dflow INFO: Task 1935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 869cc229693dc1a22d894e8bdb3988ae with result from task 1935 parsl.dataflow.dflow DEBUG: Task 1935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/d869bee9-914d-4e1d-94fe-68ff365ca7ee_isrForCrosstalkSources_1031699_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/d869bee9-914d-4e1d-94fe-68ff365ca7ee_isrForCrosstalkSources_1031699_18.stderr parsl.dataflow.dflow INFO: Task 1313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cc1dafb1f26b0e1cf1377793c9f2b49 with result from task 1313 parsl.dataflow.dflow DEBUG: Task 1313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/6a23b99e-68fe-49fe-abda-234e11ef609f_isrForCrosstalkSources_1031735_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/6a23b99e-68fe-49fe-abda-234e11ef609f_isrForCrosstalkSources_1031735_47.stderr parsl.dataflow.dflow INFO: Task 1831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1e26fd3a7166ad34597d11362923ee with result from task 1831 parsl.dataflow.dflow DEBUG: Task 1831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/92bf2cbf-5967-40e2-aeb6-fc823486f04c_isrForCrosstalkSources_1031708_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/92bf2cbf-5967-40e2-aeb6-fc823486f04c_isrForCrosstalkSources_1031708_32.stderr parsl.dataflow.dflow INFO: Task 1709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f636baf01a94792007a43fceb1dea3ed with result from task 1709 parsl.dataflow.dflow DEBUG: Task 1709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/fb1ae897-9c10-487c-93dc-e576cbc151be_isrForCrosstalkSources_1031701_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/fb1ae897-9c10-487c-93dc-e576cbc151be_isrForCrosstalkSources_1031701_27.stderr parsl.dataflow.dflow INFO: Task 1854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c494522d487ab18d7365fe0705f8afc with result from task 1854 parsl.dataflow.dflow DEBUG: Task 1854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/bcf4b9f0-28e6-46d0-bd47-8bab74382121_isrForCrosstalkSources_1031720_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/bcf4b9f0-28e6-46d0-bd47-8bab74382121_isrForCrosstalkSources_1031720_27.stderr parsl.dataflow.dflow INFO: Task 1627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5131553fe5d1f7063039807d9bb78c0 with result from task 1627 parsl.dataflow.dflow DEBUG: Task 1627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/40f24cd4-9694-424e-8c35-8d269772c232_isrForCrosstalkSources_1031715_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/40f24cd4-9694-424e-8c35-8d269772c232_isrForCrosstalkSources_1031715_50.stderr parsl.dataflow.dflow INFO: Task 1671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6153696434add7a8c250f5e99fad70d0 with result from task 1671 parsl.dataflow.dflow DEBUG: Task 1671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/460b8c4c-558d-443d-af27-c3dafdf5f384_isrForCrosstalkSources_1031699_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/460b8c4c-558d-443d-af27-c3dafdf5f384_isrForCrosstalkSources_1031699_10.stderr 2024-03-17 04:06:30 proc_lsst.multi:170 [INFO] [multi] job_id 17148127 proc_lsst.multi INFO: [multi] job_id 17148127 2024-03-17 04:06:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 126 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 126 2024-03-17 04:06:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148127 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148127 parsl.executors.status_handling DEBUG: Launched block 181 on executor multi with job ID 17148127 parsl.executors.status_handling INFO: Allocated block ID 182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-182 2024-03-17 04:06:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 2024-03-17 04:06:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 126 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 126 2024-03-17 04:06:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c315952d9228f06d97cf9b0516a02943 with result from task 1887 parsl.dataflow.dflow DEBUG: Task 1887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2622d988-79dd-44b2-8a53-41dbda1432cc_isrForCrosstalkSources_1031708_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2622d988-79dd-44b2-8a53-41dbda1432cc_isrForCrosstalkSources_1031708_13.stderr parsl.dataflow.dflow INFO: Task 1204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c86cef50c393070ddf509aa3455a8035 with result from task 1204 parsl.dataflow.dflow DEBUG: Task 1204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6269d343-257a-4906-8c38-10dcf31f26fa_isrForCrosstalkSources_1031700_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6269d343-257a-4906-8c38-10dcf31f26fa_isrForCrosstalkSources_1031700_26.stderr parsl.dataflow.dflow INFO: Task 1241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18017adebec6097051b3ee892c4075e1 with result from task 1241 parsl.dataflow.dflow DEBUG: Task 1241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/a1522970-bbd9-4f42-a8ac-3f06c69fd58f_isrForCrosstalkSources_1031716_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/a1522970-bbd9-4f42-a8ac-3f06c69fd58f_isrForCrosstalkSources_1031716_27.stderr parsl.dataflow.dflow INFO: Task 1207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe1dd660552ee1917f6da97c9283bab with result from task 1207 parsl.dataflow.dflow DEBUG: Task 1207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/3f4cb789-91db-4113-beb0-885686fac6b2_isrForCrosstalkSources_1031717_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/3f4cb789-91db-4113-beb0-885686fac6b2_isrForCrosstalkSources_1031717_25.stderr parsl.dataflow.dflow INFO: Task 1730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11a676d7a85786f46f90d377bb438392 with result from task 1730 parsl.dataflow.dflow DEBUG: Task 1730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/a1dd9605-b03e-4d78-b284-e00ace4c0a37_isrForCrosstalkSources_1031727_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/a1dd9605-b03e-4d78-b284-e00ace4c0a37_isrForCrosstalkSources_1031727_34.stderr parsl.dataflow.dflow INFO: Task 1763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77da03198852ff7dc98f836fa32ad11e with result from task 1763 parsl.dataflow.dflow DEBUG: Task 1763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/13c4179a-4f13-457f-a36e-f0c66033ea98_isrForCrosstalkSources_1031705_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/13c4179a-4f13-457f-a36e-f0c66033ea98_isrForCrosstalkSources_1031705_39.stderr 2024-03-17 04:06:30 proc_lsst.multi:170 [INFO] [multi] job_id 17148128 proc_lsst.multi INFO: [multi] job_id 17148128 2024-03-17 04:06:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 127 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 127 2024-03-17 04:06:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148128 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148128 parsl.executors.status_handling DEBUG: Launched block 182 on executor multi with job ID 17148128 parsl.executors.status_handling INFO: Allocated block ID 183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-183 2024-03-17 04:06:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 2024-03-17 04:06:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 127 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 127 2024-03-17 04:06:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55050ade6fe33f2e1e866e6a7cba9cc8 with result from task 1892 parsl.dataflow.dflow DEBUG: Task 1892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3296bd3e-3a85-4057-bc52-44a8e0f135af_isrForCrosstalkSources_1031729_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3296bd3e-3a85-4057-bc52-44a8e0f135af_isrForCrosstalkSources_1031729_62.stderr parsl.dataflow.dflow INFO: Task 1720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f6e7f7863062c7e8e3496b75436d78 with result from task 1720 parsl.dataflow.dflow DEBUG: Task 1720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/ac3be557-4379-4bbf-867a-fa620157b7d3_isrForCrosstalkSources_1031714_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/ac3be557-4379-4bbf-867a-fa620157b7d3_isrForCrosstalkSources_1031714_26.stderr parsl.dataflow.dflow INFO: Task 1237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b975036613bf30adf7c6c6e96f440fe8 with result from task 1237 parsl.dataflow.dflow DEBUG: Task 1237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/a22528dd-b849-425c-be0e-b46c71abbcb1_isrForCrosstalkSources_1031704_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/a22528dd-b849-425c-be0e-b46c71abbcb1_isrForCrosstalkSources_1031704_14.stderr parsl.dataflow.dflow INFO: Task 1223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f107fc27c820cd0721d169d023bea10b with result from task 1223 parsl.dataflow.dflow DEBUG: Task 1223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/30785a45-24d9-4fad-b8c2-8acf09b11224_isrForCrosstalkSources_1031702_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/30785a45-24d9-4fad-b8c2-8acf09b11224_isrForCrosstalkSources_1031702_25.stderr 2024-03-17 04:06:31 proc_lsst.multi:170 [INFO] [multi] job_id 17148129 proc_lsst.multi INFO: [multi] job_id 17148129 2024-03-17 04:06:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 128 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 128 2024-03-17 04:06:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148129 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148129 parsl.executors.status_handling DEBUG: Launched block 183 on executor multi with job ID 17148129 parsl.executors.status_handling INFO: Allocated block ID 184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-184 2024-03-17 04:06:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 2024-03-17 04:06:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 128 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 128 2024-03-17 04:06:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 980404b977721198ba74e27cf731eaf6 with result from task 1752 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 1752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e6f564c3-223f-40b6-9b36-dacf10141085_isrForCrosstalkSources_1031736_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e6f564c3-223f-40b6-9b36-dacf10141085_isrForCrosstalkSources_1031736_20.stderr parsl.dataflow.dflow INFO: Task 1656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2be6dfc7d5ae9fa6e724a8f22d4b7ef with result from task 1656 parsl.dataflow.dflow DEBUG: Task 1656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/3f4e8938-4688-409c-a86e-b2d1e4895049_isrForCrosstalkSources_1031715_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/3f4e8938-4688-409c-a86e-b2d1e4895049_isrForCrosstalkSources_1031715_37.stderr parsl.dataflow.dflow INFO: Task 1484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f13356dadc435e759d8bddeeca796af5 with result from task 1484 parsl.dataflow.dflow DEBUG: Task 1484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1b0b5525-48c9-47f3-bbfa-9f5365d6333a_isrForCrosstalkSources_1031733_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1b0b5525-48c9-47f3-bbfa-9f5365d6333a_isrForCrosstalkSources_1031733_31.stderr parsl.dataflow.dflow INFO: Task 1890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65618423145660d07a250ec40591fcae with result from task 1890 parsl.dataflow.dflow DEBUG: Task 1890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c9db02ca-b3cc-4f48-9d7a-c6a355fa9dfd_isrForCrosstalkSources_1031716_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c9db02ca-b3cc-4f48-9d7a-c6a355fa9dfd_isrForCrosstalkSources_1031716_12.stderr parsl.dataflow.dflow INFO: Task 1601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90b0d1d8e0326544168b59d20a24b4d1 with result from task 1601 parsl.dataflow.dflow DEBUG: Task 1601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/97f4d939-722d-4093-85ba-cf7999964f9f_isrForCrosstalkSources_1031700_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/97f4d939-722d-4093-85ba-cf7999964f9f_isrForCrosstalkSources_1031700_62.stderr parsl.dataflow.dflow INFO: Task 1249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b9b1cfc217f51823025553130e1c6fa with result from task 1249 parsl.dataflow.dflow DEBUG: Task 1249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f65b8ebc-313b-4ed9-ba4e-4d3260aef8d4_isrForCrosstalkSources_1031737_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/f65b8ebc-313b-4ed9-ba4e-4d3260aef8d4_isrForCrosstalkSources_1031737_53.stderr parsl.dataflow.dflow INFO: Task 1567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 094dc660eb5137a6cb7eec7ae69debe8 with result from task 1567 parsl.dataflow.dflow DEBUG: Task 1567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/ebe633ac-47d1-4de1-9db4-8f5f445344a5_isrForCrosstalkSources_1031721_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/ebe633ac-47d1-4de1-9db4-8f5f445344a5_isrForCrosstalkSources_1031721_44.stderr parsl.dataflow.dflow INFO: Task 1380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 969d0ccea9db7d4291bcba4a668f9e1c with result from task 1380 parsl.dataflow.dflow DEBUG: Task 1380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bc9fbc1f-b380-47a7-bcd1-2644d779d421_isrForCrosstalkSources_1031723_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bc9fbc1f-b380-47a7-bcd1-2644d779d421_isrForCrosstalkSources_1031723_9.stderr parsl.dataflow.dflow INFO: Task 1669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f158fa11670f52135802559f8c5f25d8 with result from task 1669 parsl.dataflow.dflow DEBUG: Task 1669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/10d21cf5-2579-498c-95e4-9ccb33572913_isrForCrosstalkSources_1031714_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/10d21cf5-2579-498c-95e4-9ccb33572913_isrForCrosstalkSources_1031714_18.stderr parsl.dataflow.dflow INFO: Task 1867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a356a8694f4a91b42c8f10af76d5219 with result from task 1867 parsl.dataflow.dflow DEBUG: Task 1867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b87595cd-f744-4c36-b524-447ca7109635_isrForCrosstalkSources_1031714_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b87595cd-f744-4c36-b524-447ca7109635_isrForCrosstalkSources_1031714_29.stderr 2024-03-17 04:06:31 proc_lsst.multi:170 [INFO] [multi] job_id 17148130 proc_lsst.multi INFO: [multi] job_id 17148130 2024-03-17 04:06:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 129 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 129 2024-03-17 04:06:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148130 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148130 parsl.executors.status_handling DEBUG: Launched block 184 on executor multi with job ID 17148130 parsl.executors.status_handling INFO: Allocated block ID 185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-185 2024-03-17 04:06:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 2024-03-17 04:06:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 129 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 129 2024-03-17 04:06:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac3c16c55e8641db446809ccd775e456 with result from task 1255 parsl.dataflow.dflow DEBUG: Task 1255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/f1c454ae-e577-464d-a61d-2796dfdf5d53_isrForCrosstalkSources_1031734_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/f1c454ae-e577-464d-a61d-2796dfdf5d53_isrForCrosstalkSources_1031734_49.stderr parsl.dataflow.dflow INFO: Task 1231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe88daea0e4424f2fa711b297f827e30 with result from task 1231 parsl.dataflow.dflow DEBUG: Task 1231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7202dedf-be56-4dd4-8efe-be872cbc00f1_isrForCrosstalkSources_1031702_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7202dedf-be56-4dd4-8efe-be872cbc00f1_isrForCrosstalkSources_1031702_13.stderr parsl.dataflow.dflow INFO: Task 1362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e98049930200bd84f342e34ceccfa798 with result from task 1362 parsl.dataflow.dflow DEBUG: Task 1362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/616d46cf-cbb7-47b8-a66b-e35155e71385_isrForCrosstalkSources_1031712_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/616d46cf-cbb7-47b8-a66b-e35155e71385_isrForCrosstalkSources_1031712_17.stderr parsl.dataflow.dflow INFO: Task 1254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75c3d281cd74d13086b15fe1b0ff85ae with result from task 1254 parsl.dataflow.dflow DEBUG: Task 1254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ffe3fa38-e8b4-4fbc-8e6d-b25fb9de11d6_isrForCrosstalkSources_1031737_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/ffe3fa38-e8b4-4fbc-8e6d-b25fb9de11d6_isrForCrosstalkSources_1031737_52.stderr 2024-03-17 04:06:32 proc_lsst.multi:170 [INFO] [multi] job_id 17148131 proc_lsst.multi INFO: [multi] job_id 17148131 2024-03-17 04:06:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 130 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 130 2024-03-17 04:06:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148131 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148131 parsl.executors.status_handling DEBUG: Launched block 185 on executor multi with job ID 17148131 parsl.executors.status_handling INFO: Allocated block ID 186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-186 2024-03-17 04:06:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 130 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 130 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f88690e14f446b076fdeeaa6a3be9f9 with result from task 1221 parsl.dataflow.dflow DEBUG: Task 1221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e74c570b-0254-4b5d-8ee2-7c60c2615477_isrForCrosstalkSources_1031704_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e74c570b-0254-4b5d-8ee2-7c60c2615477_isrForCrosstalkSources_1031704_33.stderr parsl.dataflow.dflow INFO: Task 1226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 285e6722c33ff1a8985a774a8a9edcb8 with result from task 1226 parsl.dataflow.dflow DEBUG: Task 1226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/fddbc86d-8069-49b8-ba9c-8e7b2ac5e1a0_isrForCrosstalkSources_1031719_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/fddbc86d-8069-49b8-ba9c-8e7b2ac5e1a0_isrForCrosstalkSources_1031719_33.stderr parsl.dataflow.dflow INFO: Task 1796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8efb7a76477c522bc6464a9b4459041 with result from task 1796 parsl.dataflow.dflow DEBUG: Task 1796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b87318ae-0912-45e2-bf54-16b157c547d7_isrForCrosstalkSources_1031718_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b87318ae-0912-45e2-bf54-16b157c547d7_isrForCrosstalkSources_1031718_40.stderr parsl.dataflow.dflow INFO: Task 1944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca4700f8a4465988a0b7453d9b08b125 with result from task 1944 parsl.dataflow.dflow DEBUG: Task 1944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/2eef6316-0f4b-47b1-9604-2540054afb96_isrForCrosstalkSources_1031705_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/2eef6316-0f4b-47b1-9604-2540054afb96_isrForCrosstalkSources_1031705_31.stderr parsl.dataflow.dflow INFO: Task 2015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5685da80daca7348611097781638e0b7 with result from task 2015 parsl.dataflow.dflow DEBUG: Task 2015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/007de0ef-a813-4f36-8041-ba16eb92e07c_isrForCrosstalkSources_1031700_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/007de0ef-a813-4f36-8041-ba16eb92e07c_isrForCrosstalkSources_1031700_47.stderr parsl.dataflow.dflow INFO: Task 1802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14ebb24c53c4416d8d2d082ec248e1e7 with result from task 1802 parsl.dataflow.dflow DEBUG: Task 1802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/92fce5e1-6be4-4735-a4f2-9e1637b33d6b_isrForCrosstalkSources_1031718_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/92fce5e1-6be4-4735-a4f2-9e1637b33d6b_isrForCrosstalkSources_1031718_11.stderr parsl.dataflow.dflow INFO: Task 1457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 097eb7da21ec2fc281604ac3bc16d1a8 with result from task 1457 parsl.dataflow.dflow DEBUG: Task 1457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c2318c6f-c86f-4b79-bdc0-787ab0b6d68f_isrForCrosstalkSources_1031722_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c2318c6f-c86f-4b79-bdc0-787ab0b6d68f_isrForCrosstalkSources_1031722_2.stderr 2024-03-17 04:06:32 proc_lsst.multi:170 [INFO] [multi] job_id 17148132 proc_lsst.multi INFO: [multi] job_id 17148132 2024-03-17 04:06:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 131 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 131 2024-03-17 04:06:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148132 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148132 parsl.executors.status_handling DEBUG: Launched block 186 on executor multi with job ID 17148132 parsl.executors.status_handling INFO: Allocated block ID 187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-187 2024-03-17 04:06:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 131 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 131 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 508f34e63c994cedffc221e60e10a2c5 with result from task 1663 parsl.dataflow.dflow DEBUG: Task 1663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/d24fa311-e22b-41eb-bf43-7403f92645c8_isrForCrosstalkSources_1031720_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/d24fa311-e22b-41eb-bf43-7403f92645c8_isrForCrosstalkSources_1031720_30.stderr parsl.dataflow.dflow INFO: Task 1754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f205e6fede5c8eaa76cb08cf6c727bc with result from task 1754 parsl.dataflow.dflow DEBUG: Task 1754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c065d253-f737-4a51-9381-3b27c505c3bc_isrForCrosstalkSources_1031732_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c065d253-f737-4a51-9381-3b27c505c3bc_isrForCrosstalkSources_1031732_22.stderr 2024-03-17 04:06:32 proc_lsst.multi:170 [INFO] [multi] job_id 17148133 proc_lsst.multi INFO: [multi] job_id 17148133 2024-03-17 04:06:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 132 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 132 2024-03-17 04:06:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148133 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148133 parsl.executors.status_handling DEBUG: Launched block 187 on executor multi with job ID 17148133 parsl.executors.status_handling INFO: Allocated block ID 188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-188 2024-03-17 04:06:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 132 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 132 2024-03-17 04:06:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93b816a6efcffdbf3a901871560db21e with result from task 1303 parsl.dataflow.dflow DEBUG: Task 1303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f6975a89-7b68-4f3b-b748-ab5c1ade366e_isrForCrosstalkSources_1031730_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f6975a89-7b68-4f3b-b748-ab5c1ade366e_isrForCrosstalkSources_1031730_26.stderr parsl.dataflow.dflow INFO: Task 1256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccc1b1bfe0d62ae96704a347f3a7c7cd with result from task 1256 parsl.dataflow.dflow DEBUG: Task 1256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/542631e4-5288-473b-9b7e-528c8b2cd60c_isrForCrosstalkSources_1031734_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/542631e4-5288-473b-9b7e-528c8b2cd60c_isrForCrosstalkSources_1031734_62.stderr parsl.dataflow.dflow INFO: Task 1650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc027dadf8a8dd552a51a1e68add0348 with result from task 1650 parsl.dataflow.dflow DEBUG: Task 1650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/21e01752-26bf-4f4d-8165-99785b3c9a94_isrForCrosstalkSources_1031731_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/21e01752-26bf-4f4d-8165-99785b3c9a94_isrForCrosstalkSources_1031731_43.stderr parsl.dataflow.dflow INFO: Task 1220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba5283c13d27626ec28cd223eafc79a2 with result from task 1220 parsl.dataflow.dflow DEBUG: Task 1220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/464a95e8-3b5b-48a7-b42c-afaaaedc8a4c_isrForCrosstalkSources_1031735_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/464a95e8-3b5b-48a7-b42c-afaaaedc8a4c_isrForCrosstalkSources_1031735_20.stderr parsl.dataflow.dflow INFO: Task 1302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d084f29361feae1472ddd82752e1d9e3 with result from task 1302 parsl.dataflow.dflow DEBUG: Task 1302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c55bf889-c50a-4600-8378-17dc79a56053_isrForCrosstalkSources_1031722_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/c55bf889-c50a-4600-8378-17dc79a56053_isrForCrosstalkSources_1031722_34.stderr parsl.dataflow.dflow INFO: Task 1306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ff1be5557383fff09db8b4a842fe95e with result from task 1306 parsl.dataflow.dflow DEBUG: Task 1306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b7ff2e10-363d-4c39-bdad-ce19a0584e97_isrForCrosstalkSources_1031718_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b7ff2e10-363d-4c39-bdad-ce19a0584e97_isrForCrosstalkSources_1031718_14.stderr parsl.dataflow.dflow INFO: Task 1613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b8e314a4327891aecc4ae9e994ed494 with result from task 1613 parsl.dataflow.dflow DEBUG: Task 1613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/bf82ef4b-4dfa-4fc8-9f07-141fe0e5b61d_isrForCrosstalkSources_1031719_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/bf82ef4b-4dfa-4fc8-9f07-141fe0e5b61d_isrForCrosstalkSources_1031719_62.stderr 2024-03-17 04:06:33 proc_lsst.multi:170 [INFO] [multi] job_id 17148134 proc_lsst.multi INFO: [multi] job_id 17148134 2024-03-17 04:06:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 133 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 133 2024-03-17 04:06:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148134 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148134 parsl.executors.status_handling DEBUG: Launched block 188 on executor multi with job ID 17148134 parsl.executors.status_handling INFO: Allocated block ID 189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-189 2024-03-17 04:06:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 2024-03-17 04:06:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 133 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 133 2024-03-17 04:06:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3de01af2c05e3bf0e5b08f5584d9440f with result from task 1298 parsl.dataflow.dflow DEBUG: Task 1298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/f846e33a-53ea-4468-86e9-51a5306cacfb_isrForCrosstalkSources_1031722_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/f846e33a-53ea-4468-86e9-51a5306cacfb_isrForCrosstalkSources_1031722_8.stderr parsl.dataflow.dflow INFO: Task 1647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a26dc7b1dff16531b5f0c79fde5fdb93 with result from task 1647 parsl.dataflow.dflow DEBUG: Task 1647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/e9302642-e9ba-461d-acbb-45e9406863d3_isrForCrosstalkSources_1031722_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/e9302642-e9ba-461d-acbb-45e9406863d3_isrForCrosstalkSources_1031722_61.stderr parsl.dataflow.dflow INFO: Task 1809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bfc86b814f95cc271d0571098845e5f with result from task 1809 parsl.dataflow.dflow DEBUG: Task 1809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e40041d6-bdcc-484d-9595-acccdc7114aa_isrForCrosstalkSources_1031704_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/e40041d6-bdcc-484d-9595-acccdc7114aa_isrForCrosstalkSources_1031704_23.stderr parsl.dataflow.dflow INFO: Task 1251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1be4269b1747669fdffc097172c55ab5 with result from task 1251 parsl.dataflow.dflow DEBUG: Task 1251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/45bcb014-1e27-4b60-ba39-6941bf2074d5_isrForCrosstalkSources_1031702_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/45bcb014-1e27-4b60-ba39-6941bf2074d5_isrForCrosstalkSources_1031702_61.stderr parsl.dataflow.dflow INFO: Task 1494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 149e7ec6fab59e120fd59dda017639f5 with result from task 1494 parsl.dataflow.dflow DEBUG: Task 1494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6a2551f3-d8e8-4a42-a906-1b552691c0cf_isrForCrosstalkSources_1031716_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6a2551f3-d8e8-4a42-a906-1b552691c0cf_isrForCrosstalkSources_1031716_61.stderr 2024-03-17 04:06:33 proc_lsst.multi:170 [INFO] [multi] job_id 17148135 proc_lsst.multi INFO: [multi] job_id 17148135 2024-03-17 04:06:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 134 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 134 2024-03-17 04:06:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148135 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148135 parsl.executors.status_handling DEBUG: Launched block 189 on executor multi with job ID 17148135 parsl.executors.status_handling INFO: Allocated block ID 190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-190 2024-03-17 04:06:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 2024-03-17 04:06:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 134 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 134 2024-03-17 04:06:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e8894c4cf2bcbc7ab43b5b202629b8a with result from task 1571 parsl.dataflow.dflow DEBUG: Task 1571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/44d7266c-30bc-4952-8d4d-fb67f1806952_isrForCrosstalkSources_1031708_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/44d7266c-30bc-4952-8d4d-fb67f1806952_isrForCrosstalkSources_1031708_30.stderr 2024-03-17 04:06:34 proc_lsst.multi:170 [INFO] [multi] job_id 17148136 proc_lsst.multi INFO: [multi] job_id 17148136 2024-03-17 04:06:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 135 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 135 2024-03-17 04:06:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148136 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148136 parsl.executors.status_handling DEBUG: Launched block 190 on executor multi with job ID 17148136 parsl.executors.status_handling INFO: Allocated block ID 191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-191 2024-03-17 04:06:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 135 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 135 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:06:34 proc_lsst.multi:170 [INFO] [multi] job_id 17148137 proc_lsst.multi INFO: [multi] job_id 17148137 2024-03-17 04:06:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 136 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 136 2024-03-17 04:06:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148137 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148137 parsl.executors.status_handling DEBUG: Launched block 191 on executor multi with job ID 17148137 parsl.executors.status_handling INFO: Allocated block ID 192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-192 2024-03-17 04:06:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 136 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 136 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05a9e93e5f9113fc79279a955bdaec2e with result from task 1307 parsl.dataflow.dflow DEBUG: Task 1307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1fb6ef36-ec72-48d1-a258-1133363d89fd_isrForCrosstalkSources_1031703_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1fb6ef36-ec72-48d1-a258-1133363d89fd_isrForCrosstalkSources_1031703_34.stderr parsl.dataflow.dflow INFO: Task 1301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab01237b34d3d1b814dae56a6b63cd50 with result from task 1301 parsl.dataflow.dflow DEBUG: Task 1301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d5e061a9-9820-40cd-876b-db8d086a29e4_isrForCrosstalkSources_1031715_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d5e061a9-9820-40cd-876b-db8d086a29e4_isrForCrosstalkSources_1031715_20.stderr parsl.dataflow.dflow INFO: Task 1816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f656544f698b1b52d38ba3219ed6408a with result from task 1816 parsl.dataflow.dflow DEBUG: Task 1816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/56f3f81d-d2bc-47cb-9869-81a572c2d8ec_isrForCrosstalkSources_1031718_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/56f3f81d-d2bc-47cb-9869-81a572c2d8ec_isrForCrosstalkSources_1031718_17.stderr 2024-03-17 04:06:34 proc_lsst.multi:170 [INFO] [multi] job_id 17148138 proc_lsst.multi INFO: [multi] job_id 17148138 2024-03-17 04:06:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 137 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 137 2024-03-17 04:06:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148138 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148138 parsl.executors.status_handling DEBUG: Launched block 192 on executor multi with job ID 17148138 parsl.executors.status_handling INFO: Allocated block ID 193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-193 2024-03-17 04:06:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 parsl.dataflow.dflow INFO: Task 1488 completed (launched -> exec_done) proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 parsl.dataflow.memoization DEBUG: Storing app cache entry 642174482e0b830211169dbd3c9aae1e with result from task 1488 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow DEBUG: Task 1488 checkpointed proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Done checkpointing 1 tasks proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 parsl.dataflow.dflow INFO: Standard output for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/61d36a3c-451f-48ee-bbb6-b2d171776b64_isrForCrosstalkSources_1031723_49.stdout proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 parsl.dataflow.dflow INFO: Standard error for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/61d36a3c-451f-48ee-bbb6-b2d171776b64_isrForCrosstalkSources_1031723_49.stderr proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 137 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 137 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86adb7158dd1c92d6196169aeb823719 with result from task 1636 parsl.dataflow.dflow DEBUG: Task 1636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/3fa6feaf-2d3e-472b-8af4-0a4dd86edb64_isrForCrosstalkSources_1031727_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/3fa6feaf-2d3e-472b-8af4-0a4dd86edb64_isrForCrosstalkSources_1031727_36.stderr parsl.dataflow.dflow INFO: Task 1685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9246f2ce9a48a8963a7c4e226cf6dc8 with result from task 1685 parsl.dataflow.dflow DEBUG: Task 1685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/043b3f37-fb00-4bb5-b712-f5a89212b7ed_isrForCrosstalkSources_1031733_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/043b3f37-fb00-4bb5-b712-f5a89212b7ed_isrForCrosstalkSources_1031733_34.stderr 2024-03-17 04:06:34 proc_lsst.multi:170 [INFO] [multi] job_id 17148139 proc_lsst.multi INFO: [multi] job_id 17148139 2024-03-17 04:06:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 138 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 138 2024-03-17 04:06:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148139 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148139 parsl.executors.status_handling DEBUG: Launched block 193 on executor multi with job ID 17148139 parsl.executors.status_handling INFO: Allocated block ID 194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-194 2024-03-17 04:06:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 parsl.dataflow.dflow INFO: Task 1582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c094e65dbf79206e38e10c9a3ad66389 with result from task 1582 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 138 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 138 2024-03-17 04:06:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 1582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2af11f85-0743-495c-a0dd-7aaaeebf692e_isrForCrosstalkSources_1031727_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/2af11f85-0743-495c-a0dd-7aaaeebf692e_isrForCrosstalkSources_1031727_47.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6757c6adb4e6a7181e52cdfcb6bb73ba with result from task 1923 parsl.dataflow.dflow DEBUG: Task 1923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/ce98becc-8bf2-4e4a-95aa-86e017124654_isrForCrosstalkSources_1031712_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/ce98becc-8bf2-4e4a-95aa-86e017124654_isrForCrosstalkSources_1031712_40.stderr parsl.dataflow.dflow INFO: Task 1819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3f93002c67f1be3e365b5417e746350 with result from task 1819 parsl.dataflow.dflow DEBUG: Task 1819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/346ca8dd-cf68-4a83-99b7-50e30afbe338_isrForCrosstalkSources_1031727_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/346ca8dd-cf68-4a83-99b7-50e30afbe338_isrForCrosstalkSources_1031727_33.stderr 2024-03-17 04:06:35 proc_lsst.multi:170 [INFO] [multi] job_id 17148140 proc_lsst.multi INFO: [multi] job_id 17148140 2024-03-17 04:06:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 139 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 139 2024-03-17 04:06:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148140 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148140 parsl.executors.status_handling DEBUG: Launched block 194 on executor multi with job ID 17148140 parsl.executors.status_handling INFO: Allocated block ID 195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-195 2024-03-17 04:06:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 parsl.dataflow.dflow INFO: Task 1233 completed (launched -> exec_done) proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 parsl.dataflow.memoization DEBUG: Storing app cache entry a70d36fe1ec6705041aad8dab7220f41 with result from task 1233 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 parsl.dataflow.dflow DEBUG: Task 1233 checkpointed proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Standard output for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/df9c587d-2e84-465c-bce2-8fdac255bc9a_isrForCrosstalkSources_1031699_13.stdout proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 parsl.dataflow.dflow INFO: Standard error for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/df9c587d-2e84-465c-bce2-8fdac255bc9a_isrForCrosstalkSources_1031699_13.stderr proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 139 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 139 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d173c178645ca7bd9e7c15ae1ae185d with result from task 1324 parsl.dataflow.dflow DEBUG: Task 1324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8e8c7867-6faf-451c-a32b-ecb127fb07d5_isrForCrosstalkSources_1031708_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/8e8c7867-6faf-451c-a32b-ecb127fb07d5_isrForCrosstalkSources_1031708_28.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6fe3ed5d6b20b392dd2b75564eafbef with result from task 1520 parsl.dataflow.dflow DEBUG: Task 1520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1788db7e-498b-4e33-80ca-bb7f29cbc1a7_isrForCrosstalkSources_1031703_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1788db7e-498b-4e33-80ca-bb7f29cbc1a7_isrForCrosstalkSources_1031703_60.stderr parsl.dataflow.dflow INFO: Task 1765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2689b9c663e2f5dd5921807f490324d2 with result from task 1765 parsl.dataflow.dflow DEBUG: Task 1765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/42f7f87a-e135-48bc-b2d6-69e7f1f5f0a3_isrForCrosstalkSources_1031699_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/42f7f87a-e135-48bc-b2d6-69e7f1f5f0a3_isrForCrosstalkSources_1031699_8.stderr 2024-03-17 04:06:35 proc_lsst.multi:170 [INFO] [multi] job_id 17148141 proc_lsst.multi INFO: [multi] job_id 17148141 2024-03-17 04:06:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 140 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 140 2024-03-17 04:06:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148141 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148141 parsl.executors.status_handling DEBUG: Launched block 195 on executor multi with job ID 17148141 parsl.executors.status_handling INFO: Allocated block ID 196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-196 2024-03-17 04:06:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 140 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 140 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92317d6a9ec6b2c776a23cc26fa825d4 with result from task 1448 parsl.dataflow.dflow DEBUG: Task 1448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/c1decf57-e3da-49a5-9444-bc5c67c4a505_isrForCrosstalkSources_1031704_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/c1decf57-e3da-49a5-9444-bc5c67c4a505_isrForCrosstalkSources_1031704_29.stderr parsl.dataflow.dflow INFO: Task 1929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a503283fa4631c1e86ef35c81e5d768a with result from task 1929 parsl.dataflow.dflow DEBUG: Task 1929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/731d8a82-db21-4e97-98ef-08fefec70424_isrForCrosstalkSources_1031728_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/731d8a82-db21-4e97-98ef-08fefec70424_isrForCrosstalkSources_1031728_50.stderr parsl.dataflow.dflow INFO: Task 1383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6652218a062541f4bcb213ed2d803f26 with result from task 1383 parsl.dataflow.dflow DEBUG: Task 1383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/be4c5cdc-85bc-4f7c-b74d-88fc65a35588_isrForCrosstalkSources_1031702_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/be4c5cdc-85bc-4f7c-b74d-88fc65a35588_isrForCrosstalkSources_1031702_12.stderr 2024-03-17 04:06:35 proc_lsst.multi:170 [INFO] [multi] job_id 17148142 proc_lsst.multi INFO: [multi] job_id 17148142 2024-03-17 04:06:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 141 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 141 2024-03-17 04:06:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148142 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148142 parsl.executors.status_handling DEBUG: Launched block 196 on executor multi with job ID 17148142 parsl.executors.status_handling INFO: Allocated block ID 197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-197 2024-03-17 04:06:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 141 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 141 2024-03-17 04:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adfd46425d4484b84f78e3902572ec0d with result from task 1328 parsl.dataflow.dflow DEBUG: Task 1328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/4a323272-dbc0-4cbb-a201-623b539beaf9_isrForCrosstalkSources_1031700_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/4a323272-dbc0-4cbb-a201-623b539beaf9_isrForCrosstalkSources_1031700_59.stderr parsl.dataflow.dflow INFO: Task 1861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a700450cc5440055442240689ca09872 with result from task 1861 parsl.dataflow.dflow DEBUG: Task 1861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c2603019-ec6d-469d-b4a9-fd3a5e3162dd_isrForCrosstalkSources_1031716_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c2603019-ec6d-469d-b4a9-fd3a5e3162dd_isrForCrosstalkSources_1031716_41.stderr parsl.dataflow.dflow INFO: Task 1319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11c229444516f93bff858d8d07529eed with result from task 1319 parsl.dataflow.dflow DEBUG: Task 1319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/e198b303-35b6-4dee-9e5a-66105e08a110_isrForCrosstalkSources_1031702_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/e198b303-35b6-4dee-9e5a-66105e08a110_isrForCrosstalkSources_1031702_43.stderr 2024-03-17 04:06:36 proc_lsst.multi:170 [INFO] [multi] job_id 17148143 proc_lsst.multi INFO: [multi] job_id 17148143 2024-03-17 04:06:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 142 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 142 2024-03-17 04:06:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148143 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148143 parsl.executors.status_handling DEBUG: Launched block 197 on executor multi with job ID 17148143 parsl.executors.status_handling INFO: Allocated block ID 198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-198 2024-03-17 04:06:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 2024-03-17 04:06:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Task 2059 completed (launched -> exec_done) proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 142 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 142 2024-03-17 04:06:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.memoization DEBUG: Storing app cache entry 87f82aec12df5571584a0c02164b3261 with result from task 2059 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 2059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/d36b3917-a9b5-4b82-9a99-d05f2a599e6c_isrForCrosstalkSources_1031712_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/d36b3917-a9b5-4b82-9a99-d05f2a599e6c_isrForCrosstalkSources_1031712_53.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10ea28955ff01f40b90948e365611b2f with result from task 1924 parsl.dataflow.dflow DEBUG: Task 1924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4ce9c7bd-1497-4359-8c69-69d36617de01_isrForCrosstalkSources_1031699_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/4ce9c7bd-1497-4359-8c69-69d36617de01_isrForCrosstalkSources_1031699_51.stderr parsl.dataflow.dflow INFO: Task 1365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0c08faa984e23615727aa6330fc5efc with result from task 1365 parsl.dataflow.dflow DEBUG: Task 1365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/936d5087-82e9-4fec-96fe-0d7de67311de_isrForCrosstalkSources_1031731_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/936d5087-82e9-4fec-96fe-0d7de67311de_isrForCrosstalkSources_1031731_38.stderr parsl.dataflow.dflow INFO: Task 2067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beae7ab5216c518e000f74c3cb461881 with result from task 2067 parsl.dataflow.dflow DEBUG: Task 2067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/354f9fe3-481a-470b-9c19-e3bf3e848259_isrForCrosstalkSources_1031699_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/354f9fe3-481a-470b-9c19-e3bf3e848259_isrForCrosstalkSources_1031699_52.stderr parsl.dataflow.dflow INFO: Task 2083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17cbf04c20eb8b439b19434a8dbbd7e3 with result from task 2083 parsl.dataflow.dflow DEBUG: Task 2083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/b9986e23-e533-43fc-87ff-04ff20b93b0c_isrForCrosstalkSources_1031736_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/b9986e23-e533-43fc-87ff-04ff20b93b0c_isrForCrosstalkSources_1031736_6.stderr 2024-03-17 04:06:37 proc_lsst.multi:170 [INFO] [multi] job_id 17148144 proc_lsst.multi INFO: [multi] job_id 17148144 2024-03-17 04:06:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 143 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 143 2024-03-17 04:06:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148144 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148144 parsl.executors.status_handling DEBUG: Launched block 198 on executor multi with job ID 17148144 parsl.executors.status_handling INFO: Allocated block ID 199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-199 2024-03-17 04:06:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 2024-03-17 04:06:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 143 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 143 2024-03-17 04:06:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4beef8ae379de2211bb5a7e73e56157 with result from task 2054 parsl.dataflow.dflow DEBUG: Task 2054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a2244f1b-2d5f-4770-a277-b116569d1efd_isrForCrosstalkSources_1031712_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a2244f1b-2d5f-4770-a277-b116569d1efd_isrForCrosstalkSources_1031712_30.stderr parsl.dataflow.dflow INFO: Task 1367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7259b1ecfe0525a641420bfe3c1fd515 with result from task 1367 parsl.dataflow.dflow DEBUG: Task 1367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d1ad88f7-bdcc-4c61-99ab-7a801039ae2c_isrForCrosstalkSources_1031730_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d1ad88f7-bdcc-4c61-99ab-7a801039ae2c_isrForCrosstalkSources_1031730_31.stderr parsl.dataflow.dflow INFO: Task 1922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa11725271286e73f2b7d98ce69e6ef with result from task 1922 parsl.dataflow.dflow DEBUG: Task 1922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a12848d7-1ea7-4c68-b058-81552877f56e_isrForCrosstalkSources_1031723_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a12848d7-1ea7-4c68-b058-81552877f56e_isrForCrosstalkSources_1031723_58.stderr parsl.dataflow.dflow INFO: Task 2099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ff4c2da81639dd95c920a9d0b2d183f with result from task 2099 parsl.dataflow.dflow DEBUG: Task 2099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/83325b05-ce75-41a0-868d-93b94a2e2e22_isrForCrosstalkSources_1031725_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/83325b05-ce75-41a0-868d-93b94a2e2e22_isrForCrosstalkSources_1031725_32.stderr parsl.dataflow.dflow INFO: Task 1167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e199027f94782809b59ecdbdef4a66eb with result from task 1167 parsl.dataflow.dflow DEBUG: Task 1167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9aae791b-33eb-4096-96d7-904fa8d909f2_isrForCrosstalkSources_1031718_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/9aae791b-33eb-4096-96d7-904fa8d909f2_isrForCrosstalkSources_1031718_34.stderr parsl.dataflow.dflow INFO: Task 1164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db82370995e05ad06e1dd361981d2e1 with result from task 1164 parsl.dataflow.dflow DEBUG: Task 1164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/5ba37f56-9c38-413f-bd7a-068d408444b3_isrForCrosstalkSources_1031725_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/5ba37f56-9c38-413f-bd7a-068d408444b3_isrForCrosstalkSources_1031725_6.stderr parsl.dataflow.dflow INFO: Task 1165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b2fca4951313a1a110ab7b7d2f47e30 with result from task 1165 parsl.dataflow.dflow DEBUG: Task 1165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f1fdb75c-4e86-46e9-945a-1eaf02a2b741_isrForCrosstalkSources_1031709_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/f1fdb75c-4e86-46e9-945a-1eaf02a2b741_isrForCrosstalkSources_1031709_33.stderr parsl.dataflow.dflow INFO: Task 1170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a9c818592b3b9e9c638f57d7fc5d008 with result from task 1170 parsl.dataflow.dflow DEBUG: Task 1170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3f8e18c5-9ef9-4c98-8c74-78dffff4ca02_isrForCrosstalkSources_1031738_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3f8e18c5-9ef9-4c98-8c74-78dffff4ca02_isrForCrosstalkSources_1031738_56.stderr parsl.dataflow.dflow INFO: Task 1166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccb4341449cabcfcc592dbf65d77033a with result from task 1166 parsl.dataflow.dflow DEBUG: Task 1166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/9ec0b6a0-2ed4-4d81-beaf-de59b2ee4297_isrForCrosstalkSources_1031737_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/9ec0b6a0-2ed4-4d81-beaf-de59b2ee4297_isrForCrosstalkSources_1031737_27.stderr parsl.dataflow.dflow INFO: Task 1171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 791e418e376d38b289db9a94d8dc4ed2 with result from task 1171 parsl.dataflow.dflow DEBUG: Task 1171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d88df21e-112e-4518-8868-62cbf4eb05dd_isrForCrosstalkSources_1031738_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d88df21e-112e-4518-8868-62cbf4eb05dd_isrForCrosstalkSources_1031738_60.stderr parsl.dataflow.dflow INFO: Task 1153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc25012eb16b693d4a226047e2397e38 with result from task 1153 parsl.dataflow.dflow DEBUG: Task 1153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/3e1a216d-d118-4f5d-8f2e-cc8a14b453ac_isrForCrosstalkSources_1031701_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/3e1a216d-d118-4f5d-8f2e-cc8a14b453ac_isrForCrosstalkSources_1031701_11.stderr parsl.dataflow.dflow INFO: Task 1155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19fcc990062dcf3007bb5a7b58e69d84 with result from task 1155 parsl.dataflow.dflow DEBUG: Task 1155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/2fe3007c-e543-4525-92b3-b4677e850574_isrForCrosstalkSources_1031725_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/2fe3007c-e543-4525-92b3-b4677e850574_isrForCrosstalkSources_1031725_13.stderr parsl.dataflow.dflow INFO: Task 1157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 860030ff8628e5bda6c476d6b7aac879 with result from task 1157 parsl.dataflow.dflow DEBUG: Task 1157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/260bcf4c-6c4f-4e86-b5d5-b2d887d03a72_isrForCrosstalkSources_1031738_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/260bcf4c-6c4f-4e86-b5d5-b2d887d03a72_isrForCrosstalkSources_1031738_42.stderr parsl.dataflow.dflow INFO: Task 1158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 099e16dc05b1bdd0b360318e0b26717b with result from task 1158 parsl.dataflow.dflow DEBUG: Task 1158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5cd30635-b101-409d-9610-fbd9f71149af_isrForCrosstalkSources_1031735_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5cd30635-b101-409d-9610-fbd9f71149af_isrForCrosstalkSources_1031735_38.stderr parsl.dataflow.dflow INFO: Task 1154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fb79c61aa9a8215e8c744ac7aabf9be with result from task 1154 parsl.dataflow.dflow DEBUG: Task 1154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/cc8e580e-e698-4282-9575-29eee14aba62_isrForCrosstalkSources_1031735_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/cc8e580e-e698-4282-9575-29eee14aba62_isrForCrosstalkSources_1031735_7.stderr parsl.dataflow.dflow INFO: Task 1169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 543dcc4b04bc365a292543215e69ebca with result from task 1169 parsl.dataflow.dflow DEBUG: Task 1169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/097df3b1-04fa-4f6a-afd1-9b5dc7ce137b_isrForCrosstalkSources_1031710_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/097df3b1-04fa-4f6a-afd1-9b5dc7ce137b_isrForCrosstalkSources_1031710_29.stderr parsl.dataflow.dflow INFO: Task 1156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d933fbfc52aa3be707122b6a633ffdc with result from task 1156 parsl.dataflow.dflow DEBUG: Task 1156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e28a358e-99f5-4d6b-ab00-acdb04200319_isrForCrosstalkSources_1031706_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e28a358e-99f5-4d6b-ab00-acdb04200319_isrForCrosstalkSources_1031706_30.stderr parsl.dataflow.dflow INFO: Task 1152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e73abd06d3cb78b2d3b9077cb298180b with result from task 1152 parsl.dataflow.dflow DEBUG: Task 1152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/b409c3ea-d581-478a-8bdd-e0acfec25225_isrForCrosstalkSources_1031717_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/b409c3ea-d581-478a-8bdd-e0acfec25225_isrForCrosstalkSources_1031717_29.stderr parsl.dataflow.dflow INFO: Task 1159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9eb06ead68703c80a56c17251dd7ede with result from task 1159 parsl.dataflow.dflow DEBUG: Task 1159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/be5be05d-0a48-420f-bfb6-72c021ea5e06_isrForCrosstalkSources_1031702_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/be5be05d-0a48-420f-bfb6-72c021ea5e06_isrForCrosstalkSources_1031702_2.stderr 2024-03-17 04:06:37 proc_lsst.multi:170 [INFO] [multi] job_id 17148145 proc_lsst.multi INFO: [multi] job_id 17148145 2024-03-17 04:06:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 144 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 144 2024-03-17 04:06:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148145 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148145 parsl.executors.status_handling DEBUG: Launched block 199 on executor multi with job ID 17148145 parsl.executors.status_handling INFO: Allocated block ID 200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-200 2024-03-17 04:06:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 2024-03-17 04:06:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 144 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 144 2024-03-17 04:06:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb2659d7f04d885bb5da448e78706560 with result from task 1168 parsl.dataflow.dflow DEBUG: Task 1168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/133fd46b-0eeb-4b82-970d-c8809585f2b5_isrForCrosstalkSources_1031731_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/133fd46b-0eeb-4b82-970d-c8809585f2b5_isrForCrosstalkSources_1031731_50.stderr parsl.dataflow.dflow INFO: Task 1238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f9b152c90ac7ba9e2d8b024c3347d97 with result from task 1238 parsl.dataflow.dflow DEBUG: Task 1238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/bd80fefe-efb8-45ef-98db-840525a7c014_isrForCrosstalkSources_1031731_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/bd80fefe-efb8-45ef-98db-840525a7c014_isrForCrosstalkSources_1031731_15.stderr parsl.dataflow.dflow INFO: Task 1329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0dd1af97ac668e7206e1e61d9bd00b6 with result from task 1329 parsl.dataflow.dflow DEBUG: Task 1329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/3d6b7c61-605e-47f0-91b4-16620cecd163_isrForCrosstalkSources_1031734_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/3d6b7c61-605e-47f0-91b4-16620cecd163_isrForCrosstalkSources_1031734_59.stderr 2024-03-17 04:06:38 proc_lsst.multi:170 [INFO] [multi] job_id 17148146 proc_lsst.multi INFO: [multi] job_id 17148146 2024-03-17 04:06:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 145 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 145 2024-03-17 04:06:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148146 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148146 parsl.executors.status_handling DEBUG: Launched block 200 on executor multi with job ID 17148146 parsl.executors.status_handling INFO: Allocated block ID 201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-201 2024-03-17 04:06:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 2024-03-17 04:06:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 145 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 145 2024-03-17 04:06:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05c653daaabe452682463fb291544d0c with result from task 1928 parsl.dataflow.dflow DEBUG: Task 1928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/54a71da7-0714-4db1-bb79-3be0357ecb57_isrForCrosstalkSources_1031714_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/54a71da7-0714-4db1-bb79-3be0357ecb57_isrForCrosstalkSources_1031714_17.stderr parsl.dataflow.dflow INFO: Task 1342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 508fa3f446ebdb2a32d5ddde356a9162 with result from task 1342 parsl.dataflow.dflow DEBUG: Task 1342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/44e05f87-d1f4-44e4-980a-c426723109e1_isrForCrosstalkSources_1031720_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/44e05f87-d1f4-44e4-980a-c426723109e1_isrForCrosstalkSources_1031720_19.stderr parsl.dataflow.dflow INFO: Task 1797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b414f09b61cbe355a92f154644d2dba4 with result from task 1797 parsl.dataflow.dflow DEBUG: Task 1797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a1cf119a-6f9f-4c89-adc3-5c46ce3732a7_isrForCrosstalkSources_1031722_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a1cf119a-6f9f-4c89-adc3-5c46ce3732a7_isrForCrosstalkSources_1031722_12.stderr parsl.dataflow.dflow INFO: Task 1769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a75c868e385b875449c5fe6fdd182f23 with result from task 1769 parsl.dataflow.dflow DEBUG: Task 1769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/73045bd4-39bd-4fca-8cbd-4c0e5138f83a_isrForCrosstalkSources_1031736_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/73045bd4-39bd-4fca-8cbd-4c0e5138f83a_isrForCrosstalkSources_1031736_12.stderr 2024-03-17 04:06:38 proc_lsst.multi:170 [INFO] [multi] job_id 17148147 proc_lsst.multi INFO: [multi] job_id 17148147 2024-03-17 04:06:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 146 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 146 2024-03-17 04:06:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148147 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148147 parsl.executors.status_handling DEBUG: Launched block 201 on executor multi with job ID 17148147 parsl.executors.status_handling INFO: Allocated block ID 202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-202 2024-03-17 04:06:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 2024-03-17 04:06:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 146 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 146 2024-03-17 04:06:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9983b522d48805b2f8ec1141ece3781 with result from task 1326 parsl.dataflow.dflow DEBUG: Task 1326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a117298f-2828-4c7d-bfcb-9c47a29fcccd_isrForCrosstalkSources_1031711_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a117298f-2828-4c7d-bfcb-9c47a29fcccd_isrForCrosstalkSources_1031711_43.stderr parsl.dataflow.dflow INFO: Task 1232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd5a1e0d202fc67b33d1316db97cf464 with result from task 1232 parsl.dataflow.dflow DEBUG: Task 1232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/c99273df-0dbc-454a-a640-4929c585e52f_isrForCrosstalkSources_1031709_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/c99273df-0dbc-454a-a640-4929c585e52f_isrForCrosstalkSources_1031709_17.stderr parsl.dataflow.dflow INFO: Task 1783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 467b0ce0b78763d501b1d9aaae52e2f9 with result from task 1783 parsl.dataflow.dflow DEBUG: Task 1783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0574eec4-5dd2-4d45-8a9b-5c7731836cb6_isrForCrosstalkSources_1031705_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0574eec4-5dd2-4d45-8a9b-5c7731836cb6_isrForCrosstalkSources_1031705_25.stderr parsl.dataflow.dflow INFO: Task 1921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2321f0e2bdd12fb8100b5b995575117a with result from task 1921 parsl.dataflow.dflow DEBUG: Task 1921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/a4c4994b-fe7b-4b3d-8d58-f84fa8c95ec4_isrForCrosstalkSources_1031714_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/a4c4994b-fe7b-4b3d-8d58-f84fa8c95ec4_isrForCrosstalkSources_1031714_19.stderr parsl.dataflow.dflow INFO: Task 1736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f3c4bcc3cc7126667ddd5eefde548e3 with result from task 1736 parsl.dataflow.dflow DEBUG: Task 1736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/aaa904b3-5763-45dc-92bb-40403cd3a72f_isrForCrosstalkSources_1031717_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/aaa904b3-5763-45dc-92bb-40403cd3a72f_isrForCrosstalkSources_1031717_39.stderr 2024-03-17 04:06:39 proc_lsst.multi:170 [INFO] [multi] job_id 17148148 proc_lsst.multi INFO: [multi] job_id 17148148 2024-03-17 04:06:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 147 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 147 2024-03-17 04:06:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148148 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148148 parsl.executors.status_handling DEBUG: Launched block 202 on executor multi with job ID 17148148 parsl.executors.status_handling INFO: Allocated block ID 203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-203 2024-03-17 04:06:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 2024-03-17 04:06:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 147 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 147 2024-03-17 04:06:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c5c598cd4c86303ca59a4e823df9410 with result from task 1364 parsl.dataflow.dflow DEBUG: Task 1364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/eee0f548-f462-4034-9ab7-623f7bd86f85_isrForCrosstalkSources_1031737_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/eee0f548-f462-4034-9ab7-623f7bd86f85_isrForCrosstalkSources_1031737_48.stderr parsl.dataflow.dflow INFO: Task 1681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e32dce065724b9c9f42386f9e29a9fb7 with result from task 1681 parsl.dataflow.dflow DEBUG: Task 1681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e22cda9c-a47f-4fda-9c57-b47a4f56b688_isrForCrosstalkSources_1031706_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e22cda9c-a47f-4fda-9c57-b47a4f56b688_isrForCrosstalkSources_1031706_10.stderr parsl.dataflow.dflow INFO: Task 1823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fa7c5dcaf97afa33747f938a1ef1229 with result from task 1823 parsl.dataflow.dflow DEBUG: Task 1823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/707d1991-3b44-48bd-9363-8f336d245168_isrForCrosstalkSources_1031717_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/707d1991-3b44-48bd-9363-8f336d245168_isrForCrosstalkSources_1031717_4.stderr parsl.dataflow.dflow INFO: Task 1387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3efc4fc276062d3620476d5c786d779 with result from task 1387 parsl.dataflow.dflow DEBUG: Task 1387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/7c87490e-43b7-4aae-8e39-c95d85ed9c1f_isrForCrosstalkSources_1031737_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/7c87490e-43b7-4aae-8e39-c95d85ed9c1f_isrForCrosstalkSources_1031737_50.stderr parsl.dataflow.dflow INFO: Task 1209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d3c112e06ecac6665efd334b0bed349 with result from task 1209 parsl.dataflow.dflow DEBUG: Task 1209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4505e8bd-d9ec-4099-a448-99d14f8e2a89_isrForCrosstalkSources_1031706_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4505e8bd-d9ec-4099-a448-99d14f8e2a89_isrForCrosstalkSources_1031706_4.stderr parsl.dataflow.dflow INFO: Task 1799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 911114f21ab024f26ac47442679bec1d with result from task 1799 parsl.dataflow.dflow DEBUG: Task 1799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ddd5b79a-08fd-4436-98e1-7dae6577d540_isrForCrosstalkSources_1031709_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ddd5b79a-08fd-4436-98e1-7dae6577d540_isrForCrosstalkSources_1031709_36.stderr parsl.dataflow.dflow INFO: Task 1469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e08b0e2eed9258824d71fa63f305c21b with result from task 1469 parsl.dataflow.dflow DEBUG: Task 1469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/1885307d-8fcc-454c-8b8a-d9a40ba04849_isrForCrosstalkSources_1031725_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/1885307d-8fcc-454c-8b8a-d9a40ba04849_isrForCrosstalkSources_1031725_58.stderr 2024-03-17 04:06:39 proc_lsst.multi:170 [INFO] [multi] job_id 17148149 proc_lsst.multi INFO: [multi] job_id 17148149 2024-03-17 04:06:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 148 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 148 2024-03-17 04:06:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148149 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148149 parsl.executors.status_handling DEBUG: Launched block 203 on executor multi with job ID 17148149 parsl.executors.status_handling INFO: Allocated block ID 204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-204 2024-03-17 04:06:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 2024-03-17 04:06:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Task 1259 completed (launched -> exec_done) proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 148 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 148 2024-03-17 04:06:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.memoization DEBUG: Storing app cache entry 9d97322e3672bce6594f9a7ad5c137e1 with result from task 1259 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 1259 checkpointed parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:06:40 proc_lsst.multi:170 [INFO] [multi] job_id 17148150 proc_lsst.multi INFO: [multi] job_id 17148150 2024-03-17 04:06:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 149 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 149 2024-03-17 04:06:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148150 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/4ec22b9a-9e81-4ac4-920a-4f3fa8b619d0_isrForCrosstalkSources_1031711_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/4ec22b9a-9e81-4ac4-920a-4f3fa8b619d0_isrForCrosstalkSources_1031711_60.stderr parsl.dataflow.dflow INFO: Task 1299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3e40fc9ed7418de22e4d0f2b640e8bb with result from task 1299 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148150 parsl.executors.status_handling DEBUG: Launched block 204 on executor multi with job ID 17148150 parsl.executors.status_handling INFO: Allocated block ID 205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-205 2024-03-17 04:06:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 2024-03-17 04:06:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 149 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 149 2024-03-17 04:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 1299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/551030c7-1d4a-478c-a1d2-c12089dc1406_isrForCrosstalkSources_1031717_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/551030c7-1d4a-478c-a1d2-c12089dc1406_isrForCrosstalkSources_1031717_10.stderr parsl.dataflow.dflow INFO: Task 1477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4aa4ce42541db11f66ab2cfea7ed5e24 with result from task 1477 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 1477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/2a025670-c3d5-4382-9a21-cca0e61776b4_isrForCrosstalkSources_1031713_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/2a025670-c3d5-4382-9a21-cca0e61776b4_isrForCrosstalkSources_1031713_60.stderr parsl.dataflow.dflow INFO: Task 1615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9047c7d8e0bd6c26f8857f0bd70105ec with result from task 1615 parsl.dataflow.dflow DEBUG: Task 1615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/30129a4c-5f41-465e-a8f3-51e5868e8193_isrForCrosstalkSources_1031724_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/30129a4c-5f41-465e-a8f3-51e5868e8193_isrForCrosstalkSources_1031724_62.stderr parsl.dataflow.dflow INFO: Task 1227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87c8753c0f5525d5ee2aa4087de95cc1 with result from task 1227 parsl.dataflow.dflow DEBUG: Task 1227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8cb8e2e4-0377-4902-89f1-de2a06ca876e_isrForCrosstalkSources_1031700_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/8cb8e2e4-0377-4902-89f1-de2a06ca876e_isrForCrosstalkSources_1031700_51.stderr parsl.dataflow.dflow INFO: Task 1667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eecd7325a84bb3dfb9f1b5d6fd9bcf45 with result from task 1667 parsl.dataflow.dflow DEBUG: Task 1667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ead96e9d-f0a6-433c-90f0-cce7ce803b54_isrForCrosstalkSources_1031713_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ead96e9d-f0a6-433c-90f0-cce7ce803b54_isrForCrosstalkSources_1031713_40.stderr parsl.dataflow.dflow INFO: Task 1698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d980893c8753413598cae422de24b34 with result from task 1698 parsl.dataflow.dflow DEBUG: Task 1698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a7333962-05d1-4451-baa0-04795cc7930d_isrForCrosstalkSources_1031720_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a7333962-05d1-4451-baa0-04795cc7930d_isrForCrosstalkSources_1031720_9.stderr parsl.dataflow.dflow INFO: Task 2106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af47e27e50065b4c55a9c9b371c24c3a with result from task 2106 parsl.dataflow.dflow DEBUG: Task 2106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/98d17e8e-a505-44b1-87c9-7686978555ba_isrForCrosstalkSources_1031713_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/98d17e8e-a505-44b1-87c9-7686978555ba_isrForCrosstalkSources_1031713_3.stderr parsl.dataflow.dflow INFO: Task 1672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7aa09f8e07452086e4dad137b38dc453 with result from task 1672 parsl.dataflow.dflow DEBUG: Task 1672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/88750516-5048-4257-bb8e-841ea8a5c2a5_isrForCrosstalkSources_1031732_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/88750516-5048-4257-bb8e-841ea8a5c2a5_isrForCrosstalkSources_1031732_26.stderr parsl.dataflow.dflow INFO: Task 1327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a69057ef037415c84a6483eecd6e0f30 with result from task 1327 parsl.dataflow.dflow DEBUG: Task 1327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/abcd5295-2a17-46aa-9f1a-15504cc6db66_isrForCrosstalkSources_1031727_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/abcd5295-2a17-46aa-9f1a-15504cc6db66_isrForCrosstalkSources_1031727_57.stderr parsl.dataflow.dflow INFO: Task 2100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bff5f791a0fa67f5b9bd247f9e711d9e with result from task 2100 parsl.dataflow.dflow DEBUG: Task 2100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/00d562b3-44dd-432b-8552-9be5981a01f0_isrForCrosstalkSources_1031710_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/00d562b3-44dd-432b-8552-9be5981a01f0_isrForCrosstalkSources_1031710_35.stderr parsl.dataflow.dflow INFO: Task 2129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9e0bffb9986b3bc15fe871e7513f34a with result from task 2129 parsl.dataflow.dflow DEBUG: Task 2129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/fa4849dc-ade4-46ca-ba74-b01c0aa8c523_isrForCrosstalkSources_1031731_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/fa4849dc-ade4-46ca-ba74-b01c0aa8c523_isrForCrosstalkSources_1031731_47.stderr parsl.dataflow.dflow INFO: Task 1325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40992000554dcebfe6c3499b9d18d424 with result from task 1325 parsl.dataflow.dflow DEBUG: Task 1325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5b2d977f-9c17-4961-9c4a-ca67af2485bc_isrForCrosstalkSources_1031717_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/5b2d977f-9c17-4961-9c4a-ca67af2485bc_isrForCrosstalkSources_1031717_59.stderr parsl.dataflow.dflow INFO: Task 1717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a80f04e073d09a3a5db9462d482bb39f with result from task 1717 parsl.dataflow.dflow DEBUG: Task 1717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/c3bcdda6-1c6e-4d8a-b02a-af62b238096d_isrForCrosstalkSources_1031720_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/c3bcdda6-1c6e-4d8a-b02a-af62b238096d_isrForCrosstalkSources_1031720_32.stderr parsl.dataflow.dflow INFO: Task 1240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c35729a3269974a98ece56b58af4182 with result from task 1240 parsl.dataflow.dflow DEBUG: Task 1240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/e0302454-d593-4893-9879-0c8086ffeaf7_isrForCrosstalkSources_1031711_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/e0302454-d593-4893-9879-0c8086ffeaf7_isrForCrosstalkSources_1031711_16.stderr 2024-03-17 04:06:40 proc_lsst.multi:170 [INFO] [multi] job_id 17148151 proc_lsst.multi INFO: [multi] job_id 17148151 2024-03-17 04:06:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 150 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 150 2024-03-17 04:06:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148151 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148151 parsl.executors.status_handling DEBUG: Launched block 205 on executor multi with job ID 17148151 parsl.executors.status_handling INFO: Allocated block ID 206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-206 2024-03-17 04:06:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 2024-03-17 04:06:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 150 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 150 2024-03-17 04:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 917510ba5c5779ad9471b99c198c1581 with result from task 1363 parsl.dataflow.dflow DEBUG: Task 1363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/75b889bd-d94d-481f-9017-6524e6fb304b_isrForCrosstalkSources_1031728_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/75b889bd-d94d-481f-9017-6524e6fb304b_isrForCrosstalkSources_1031728_48.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ebab938cf763c80ea4f2d61905e3ec5 with result from task 1381 parsl.dataflow.dflow DEBUG: Task 1381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/ccd9db2b-a72a-49fd-9c49-f44278aa617b_isrForCrosstalkSources_1031735_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/ccd9db2b-a72a-49fd-9c49-f44278aa617b_isrForCrosstalkSources_1031735_57.stderr parsl.dataflow.dflow INFO: Task 1687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2651c96e75ee79e8c8ee9b23255c7260 with result from task 1687 parsl.dataflow.dflow DEBUG: Task 1687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/486ef142-987b-471c-b9af-eedaf34adca7_isrForCrosstalkSources_1031710_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/486ef142-987b-471c-b9af-eedaf34adca7_isrForCrosstalkSources_1031710_34.stderr parsl.dataflow.dflow INFO: Task 1222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5eb5b4d0db5300eac009e24399b22d5 with result from task 1222 parsl.dataflow.dflow DEBUG: Task 1222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/9d1e2683-0207-4913-8387-c255788306da_isrForCrosstalkSources_1031708_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/9d1e2683-0207-4913-8387-c255788306da_isrForCrosstalkSources_1031708_5.stderr 2024-03-17 04:06:41 proc_lsst.multi:170 [INFO] [multi] job_id 17148152 proc_lsst.multi INFO: [multi] job_id 17148152 2024-03-17 04:06:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 151 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 151 2024-03-17 04:06:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148152 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148152 parsl.executors.status_handling DEBUG: Launched block 206 on executor multi with job ID 17148152 parsl.executors.status_handling INFO: Allocated block ID 207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-207 2024-03-17 04:06:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 2024-03-17 04:06:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 151 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 151 2024-03-17 04:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7b72ef44442daa503b2ac0d191d6a26 with result from task 1297 parsl.dataflow.dflow DEBUG: Task 1297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0518d1ca-20e0-4f7d-9dec-44979220548a_isrForCrosstalkSources_1031717_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/0518d1ca-20e0-4f7d-9dec-44979220548a_isrForCrosstalkSources_1031717_43.stderr parsl.dataflow.dflow INFO: Task 1847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cadb511e9a3769b9dbf6de6c4a19fab with result from task 1847 parsl.dataflow.dflow DEBUG: Task 1847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/eeb41e65-fc85-4d47-8a1b-b5ab96cfed45_isrForCrosstalkSources_1031722_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/eeb41e65-fc85-4d47-8a1b-b5ab96cfed45_isrForCrosstalkSources_1031722_45.stderr parsl.dataflow.dflow INFO: Task 1346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe45ada98168950f474a74f2dada4346 with result from task 1346 parsl.dataflow.dflow DEBUG: Task 1346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c756d8df-ac05-4f1a-ad69-b984305d3812_isrForCrosstalkSources_1031715_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c756d8df-ac05-4f1a-ad69-b984305d3812_isrForCrosstalkSources_1031715_22.stderr parsl.dataflow.dflow INFO: Task 1680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24c22f46062a2b8abcfb664205227d67 with result from task 1680 parsl.dataflow.dflow DEBUG: Task 1680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/0c75f447-19cc-4b69-9eb2-61fe6ee7fa44_isrForCrosstalkSources_1031715_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/0c75f447-19cc-4b69-9eb2-61fe6ee7fa44_isrForCrosstalkSources_1031715_13.stderr parsl.dataflow.dflow INFO: Task 2144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df0d2c43077c785d90ab174c1dd1167c with result from task 2144 parsl.dataflow.dflow DEBUG: Task 2144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8dd7d03e-1862-4297-93a6-fccc6c69871a_isrForCrosstalkSources_1031736_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8dd7d03e-1862-4297-93a6-fccc6c69871a_isrForCrosstalkSources_1031736_3.stderr 2024-03-17 04:06:41 proc_lsst.multi:170 [INFO] [multi] job_id 17148153 proc_lsst.multi INFO: [multi] job_id 17148153 2024-03-17 04:06:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 152 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 152 2024-03-17 04:06:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148153 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148153 parsl.executors.status_handling DEBUG: Launched block 207 on executor multi with job ID 17148153 parsl.executors.status_handling INFO: Allocated block ID 208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-208 2024-03-17 04:06:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 2024-03-17 04:06:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 152 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 152 2024-03-17 04:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e732ace353b86a4c3c0a8f2be65b0da with result from task 1740 parsl.dataflow.dflow DEBUG: Task 1740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/9cb4cfc9-bdc7-4655-95b5-f3b0bc6c6375_isrForCrosstalkSources_1031723_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/9cb4cfc9-bdc7-4655-95b5-f3b0bc6c6375_isrForCrosstalkSources_1031723_18.stderr parsl.dataflow.dflow INFO: Task 1311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5de8a90c96f2b370ab8874cd29b4f39f with result from task 1311 parsl.dataflow.dflow DEBUG: Task 1311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b18f8939-5c2f-4d5a-8801-f963076fa5a6_isrForCrosstalkSources_1031714_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b18f8939-5c2f-4d5a-8801-f963076fa5a6_isrForCrosstalkSources_1031714_7.stderr parsl.dataflow.dflow INFO: Task 1649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a4ccdea5a142e5a0b93ef911185c557 with result from task 1649 parsl.dataflow.dflow DEBUG: Task 1649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/93956866-2ac1-4cbc-a042-d6d99bdbcefa_isrForCrosstalkSources_1031708_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/93956866-2ac1-4cbc-a042-d6d99bdbcefa_isrForCrosstalkSources_1031708_19.stderr parsl.dataflow.dflow INFO: Task 2109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b64ecd92619bca12c69d119401544f38 with result from task 2109 parsl.dataflow.dflow DEBUG: Task 2109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2d222ea4-973e-4b3f-9895-4061edec616b_isrForCrosstalkSources_1031724_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/2d222ea4-973e-4b3f-9895-4061edec616b_isrForCrosstalkSources_1031724_49.stderr 2024-03-17 04:06:42 proc_lsst.multi:170 [INFO] [multi] job_id 17148154 proc_lsst.multi INFO: [multi] job_id 17148154 2024-03-17 04:06:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 153 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 153 2024-03-17 04:06:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148154 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148154 parsl.executors.status_handling DEBUG: Launched block 208 on executor multi with job ID 17148154 parsl.executors.status_handling INFO: Allocated block ID 209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-209 2024-03-17 04:06:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 2024-03-17 04:06:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 153 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 153 2024-03-17 04:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8dea76df2d5a0ab9c1a9420f176a035 with result from task 1697 parsl.dataflow.dflow DEBUG: Task 1697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/5cbe7861-aa9d-4938-8dbd-5aad51ec2be8_isrForCrosstalkSources_1031715_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/5cbe7861-aa9d-4938-8dbd-5aad51ec2be8_isrForCrosstalkSources_1031715_43.stderr parsl.dataflow.dflow INFO: Task 1282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a9fdcd87a4ae226ffe1e521d7204bf5 with result from task 1282 parsl.dataflow.dflow DEBUG: Task 1282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/4f90df70-0129-42fe-82a5-d1eabe9b983b_isrForCrosstalkSources_1031736_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/4f90df70-0129-42fe-82a5-d1eabe9b983b_isrForCrosstalkSources_1031736_59.stderr parsl.dataflow.dflow INFO: Task 1281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfbe8199e7805575a9d1ec972199eb7e with result from task 1281 parsl.dataflow.dflow DEBUG: Task 1281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4abf6fbb-c52b-46e0-a53f-a7b77eda992a_isrForCrosstalkSources_1031703_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/4abf6fbb-c52b-46e0-a53f-a7b77eda992a_isrForCrosstalkSources_1031703_47.stderr parsl.dataflow.dflow INFO: Task 1279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82a9019d045e3f9c875fac124a8e4458 with result from task 1279 parsl.dataflow.dflow DEBUG: Task 1279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/7e3ace4c-c10b-408a-b345-0a04c6680bc9_isrForCrosstalkSources_1031709_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/7e3ace4c-c10b-408a-b345-0a04c6680bc9_isrForCrosstalkSources_1031709_31.stderr parsl.dataflow.dflow INFO: Task 1276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4560a8fb8abcc4a325cbe107f4965bad with result from task 1276 parsl.dataflow.dflow DEBUG: Task 1276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d39c8fb9-b275-4fb3-81c0-42a15f9cd265_isrForCrosstalkSources_1031709_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d39c8fb9-b275-4fb3-81c0-42a15f9cd265_isrForCrosstalkSources_1031709_53.stderr parsl.dataflow.dflow INFO: Task 1272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f481617a735f4448a8773100c33afb66 with result from task 1272 parsl.dataflow.dflow DEBUG: Task 1272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/402e6c20-a9d4-481e-bd15-24dd319e2f27_isrForCrosstalkSources_1031728_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/402e6c20-a9d4-481e-bd15-24dd319e2f27_isrForCrosstalkSources_1031728_24.stderr parsl.dataflow.dflow INFO: Task 1270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c83d67337f3ce065ccbebfbbd6f42ac with result from task 1270 parsl.dataflow.dflow DEBUG: Task 1270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/cb724ebe-85c7-491a-84dc-9eed44c4e720_isrForCrosstalkSources_1031723_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/cb724ebe-85c7-491a-84dc-9eed44c4e720_isrForCrosstalkSources_1031723_59.stderr parsl.dataflow.dflow INFO: Task 1283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 632dffb5a91258e8aabac32ebb0b7de0 with result from task 1283 parsl.dataflow.dflow DEBUG: Task 1283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/bd63992f-2e1e-42d5-88b8-3a4300425968_isrForCrosstalkSources_1031719_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/bd63992f-2e1e-42d5-88b8-3a4300425968_isrForCrosstalkSources_1031719_44.stderr parsl.dataflow.dflow INFO: Task 1280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec554171ed8935a920165b23c978c7a2 with result from task 1280 parsl.dataflow.dflow DEBUG: Task 1280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4fd92a0c-3a81-4a29-9596-0468aad29ee0_isrForCrosstalkSources_1031733_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/4fd92a0c-3a81-4a29-9596-0468aad29ee0_isrForCrosstalkSources_1031733_36.stderr parsl.dataflow.dflow INFO: Task 1344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c76bc8e585d8198fb6147f5ae594f15c with result from task 1344 parsl.dataflow.dflow DEBUG: Task 1344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ec9f0843-1034-49c7-8e18-7aaddc8b229e_isrForCrosstalkSources_1031724_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ec9f0843-1034-49c7-8e18-7aaddc8b229e_isrForCrosstalkSources_1031724_51.stderr parsl.dataflow.dflow INFO: Task 1264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f8f48a87f8f15212dac3ff1130f3d6e with result from task 1264 parsl.dataflow.dflow DEBUG: Task 1264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5a3a9a8e-6bf5-4d49-b556-275d9389a5b1_isrForCrosstalkSources_1031728_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5a3a9a8e-6bf5-4d49-b556-275d9389a5b1_isrForCrosstalkSources_1031728_35.stderr parsl.dataflow.dflow INFO: Task 1286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 102ea08450c6a21c4991cbcf9a020bf5 with result from task 1286 parsl.dataflow.dflow DEBUG: Task 1286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/7962f207-73c9-485d-b498-edcf4105cd3e_isrForCrosstalkSources_1031703_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/7962f207-73c9-485d-b498-edcf4105cd3e_isrForCrosstalkSources_1031703_12.stderr parsl.dataflow.dflow INFO: Task 1271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 789df8e6f97981cd697952e82bd9a497 with result from task 1271 parsl.dataflow.dflow DEBUG: Task 1271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/41d25e2e-b09d-4f2f-9441-b3052a9d769c_isrForCrosstalkSources_1031721_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/41d25e2e-b09d-4f2f-9441-b3052a9d769c_isrForCrosstalkSources_1031721_58.stderr parsl.dataflow.dflow INFO: Task 1263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c87f9774f8c805e51279c5e9bc151fd with result from task 1263 parsl.dataflow.dflow DEBUG: Task 1263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/6465ecd8-efea-4bc5-ba64-ffe0ec3191ce_isrForCrosstalkSources_1031704_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/6465ecd8-efea-4bc5-ba64-ffe0ec3191ce_isrForCrosstalkSources_1031704_26.stderr parsl.dataflow.dflow INFO: Task 1269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25e79368e6dac1e78862d851a6bad60b with result from task 1269 parsl.dataflow.dflow DEBUG: Task 1269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e91f427c-0891-4a19-b2ce-90bec49b5f1c_isrForCrosstalkSources_1031707_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e91f427c-0891-4a19-b2ce-90bec49b5f1c_isrForCrosstalkSources_1031707_38.stderr parsl.dataflow.dflow INFO: Task 1267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5dfbdf769bec2f6a6061db9718ef88c with result from task 1267 parsl.dataflow.dflow DEBUG: Task 1267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/4fd6c007-d2a4-4c1f-a6f7-0ba7d699ef0e_isrForCrosstalkSources_1031735_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/4fd6c007-d2a4-4c1f-a6f7-0ba7d699ef0e_isrForCrosstalkSources_1031735_59.stderr parsl.dataflow.dflow INFO: Task 1278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 951a0d265475a42c87bfaa733808486a with result from task 1278 parsl.dataflow.dflow DEBUG: Task 1278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1370879e-5d01-47b8-95ec-1f8e9e6b926e_isrForCrosstalkSources_1031715_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/1370879e-5d01-47b8-95ec-1f8e9e6b926e_isrForCrosstalkSources_1031715_15.stderr parsl.dataflow.dflow INFO: Task 1273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d674dd54a36f64ba0239999e32f9041 with result from task 1273 parsl.dataflow.dflow DEBUG: Task 1273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/d770c088-e78c-43f9-be66-211538a53029_isrForCrosstalkSources_1031712_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/d770c088-e78c-43f9-be66-211538a53029_isrForCrosstalkSources_1031712_28.stderr parsl.dataflow.dflow INFO: Task 1801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d282d74c4e83dfa6a7ec30c365609a6 with result from task 1801 parsl.dataflow.dflow DEBUG: Task 1801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/462fadd5-bab7-4274-aba4-659637430386_isrForCrosstalkSources_1031738_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/462fadd5-bab7-4274-aba4-659637430386_isrForCrosstalkSources_1031738_30.stderr parsl.dataflow.dflow INFO: Task 1268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 702441d3c3280df77af565b6ebf6d85d with result from task 1268 parsl.dataflow.dflow DEBUG: Task 1268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/fefae2f9-980f-450c-ae66-0098153ea77b_isrForCrosstalkSources_1031716_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/fefae2f9-980f-450c-ae66-0098153ea77b_isrForCrosstalkSources_1031716_29.stderr parsl.dataflow.dflow INFO: Task 1285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0499fa23aa41254a5b9436748a2cad7a with result from task 1285 parsl.dataflow.dflow DEBUG: Task 1285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a4136a3d-7a9d-4f8f-bc72-e872c1e933a3_isrForCrosstalkSources_1031722_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/a4136a3d-7a9d-4f8f-bc72-e872c1e933a3_isrForCrosstalkSources_1031722_42.stderr parsl.dataflow.dflow INFO: Task 1265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11ad87d8d1df226f34da63f948e88e9f with result from task 1265 parsl.dataflow.dflow DEBUG: Task 1265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a20f1158-e531-4105-b154-7d6de93767bf_isrForCrosstalkSources_1031718_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a20f1158-e531-4105-b154-7d6de93767bf_isrForCrosstalkSources_1031718_19.stderr parsl.dataflow.dflow INFO: Task 2143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e84e1d1cb1c78866b6da54b7fa563a62 with result from task 2143 parsl.dataflow.dflow DEBUG: Task 2143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/112b4d84-bebb-4459-ac22-6a2eca1ed929_isrForCrosstalkSources_1031715_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/112b4d84-bebb-4459-ac22-6a2eca1ed929_isrForCrosstalkSources_1031715_29.stderr parsl.dataflow.dflow INFO: Task 1266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d07e4e22cae680e38b43214e422334aa with result from task 1266 parsl.dataflow.dflow DEBUG: Task 1266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/aff86859-62f6-418a-b541-4bfdf1cf2393_isrForCrosstalkSources_1031718_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/aff86859-62f6-418a-b541-4bfdf1cf2393_isrForCrosstalkSources_1031718_13.stderr parsl.dataflow.dflow INFO: Task 1277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7fb72d6d14e0940ce60edfd9ee49389 with result from task 1277 parsl.dataflow.dflow DEBUG: Task 1277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/9205921a-2b8b-4dfb-90ad-2a0d967d8035_isrForCrosstalkSources_1031705_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/9205921a-2b8b-4dfb-90ad-2a0d967d8035_isrForCrosstalkSources_1031705_26.stderr parsl.dataflow.dflow INFO: Task 1284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 222b0d8c9efd75d3fbaf681512bbad8b with result from task 1284 parsl.dataflow.dflow DEBUG: Task 1284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6de02fcb-493b-4409-a687-42c311124fdd_isrForCrosstalkSources_1031716_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/6de02fcb-493b-4409-a687-42c311124fdd_isrForCrosstalkSources_1031716_9.stderr parsl.dataflow.dflow INFO: Task 1275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b122db7eea856201896305bd7e980fb5 with result from task 1275 parsl.dataflow.dflow DEBUG: Task 1275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/310de23e-295c-48d1-879a-12369642499b_isrForCrosstalkSources_1031714_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/310de23e-295c-48d1-879a-12369642499b_isrForCrosstalkSources_1031714_4.stderr parsl.dataflow.dflow INFO: Task 1810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d566d2453125d3af54d94cb4ebd2cb34 with result from task 1810 parsl.dataflow.dflow DEBUG: Task 1810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/fbd92de9-292e-4aa8-ab2c-d6337dfc21ce_isrForCrosstalkSources_1031711_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/fbd92de9-292e-4aa8-ab2c-d6337dfc21ce_isrForCrosstalkSources_1031711_30.stderr parsl.dataflow.dflow INFO: Task 1244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c142d0ab6108e571806c6e8060a7975 with result from task 1244 parsl.dataflow.dflow DEBUG: Task 1244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/aecc549c-53cf-4b3e-a7b2-9106702db5b0_isrForCrosstalkSources_1031721_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/aecc549c-53cf-4b3e-a7b2-9106702db5b0_isrForCrosstalkSources_1031721_7.stderr parsl.dataflow.dflow INFO: Task 1274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c89082e500297780f2cceacab194c46 with result from task 1274 parsl.dataflow.dflow DEBUG: Task 1274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/24e1abc0-832f-42ff-b576-1fb70ffbf828_isrForCrosstalkSources_1031713_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/24e1abc0-832f-42ff-b576-1fb70ffbf828_isrForCrosstalkSources_1031713_22.stderr 2024-03-17 04:06:42 proc_lsst.multi:170 [INFO] [multi] job_id 17148155 proc_lsst.multi INFO: [multi] job_id 17148155 2024-03-17 04:06:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 154 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 154 2024-03-17 04:06:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148155 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148155 parsl.executors.status_handling DEBUG: Launched block 209 on executor multi with job ID 17148155 parsl.executors.status_handling INFO: Allocated block ID 210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-210 2024-03-17 04:06:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 2024-03-17 04:06:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 154 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 154 2024-03-17 04:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 296965e77486c44f5b4bc3ff2aae90b8 with result from task 1834 parsl.dataflow.dflow DEBUG: Task 1834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/6a5edb42-7861-42f1-b20c-e611ab73dfe5_isrForCrosstalkSources_1031730_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/6a5edb42-7861-42f1-b20c-e611ab73dfe5_isrForCrosstalkSources_1031730_50.stderr parsl.dataflow.dflow INFO: Task 1675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1099cbe0fac616dac3272e5f69450dd with result from task 1675 parsl.dataflow.dflow DEBUG: Task 1675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/fcb0879a-2fcb-4112-a7d8-fe7963c3acb3_isrForCrosstalkSources_1031712_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/fcb0879a-2fcb-4112-a7d8-fe7963c3acb3_isrForCrosstalkSources_1031712_25.stderr parsl.dataflow.dflow INFO: Task 1744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 714d3e589e7d16e92234cae1afa4f353 with result from task 1744 parsl.dataflow.dflow DEBUG: Task 1744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/939794d6-4efd-4c10-bad8-ffe1ec56f5b2_isrForCrosstalkSources_1031737_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/939794d6-4efd-4c10-bad8-ffe1ec56f5b2_isrForCrosstalkSources_1031737_22.stderr parsl.dataflow.dflow INFO: Task 1869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6189dcc0c56b29e898522141b75fd2c with result from task 1869 parsl.dataflow.dflow DEBUG: Task 1869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/8f26b0bb-06e5-4c49-ac18-128d46ae55a9_isrForCrosstalkSources_1031723_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/8f26b0bb-06e5-4c49-ac18-128d46ae55a9_isrForCrosstalkSources_1031723_7.stderr parsl.dataflow.dflow INFO: Task 1913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d723223cf1046fce1697c4bfbbb43be with result from task 1913 parsl.dataflow.dflow DEBUG: Task 1913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/e7e6e2e6-8a1f-4917-adba-338796686fb8_isrForCrosstalkSources_1031734_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/e7e6e2e6-8a1f-4917-adba-338796686fb8_isrForCrosstalkSources_1031734_35.stderr parsl.dataflow.dflow INFO: Task 1229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a68adf1c6bf5c6ae953621ffed184d0 with result from task 1229 parsl.dataflow.dflow DEBUG: Task 1229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/1b637027-1fab-41c8-a203-a0aa9c998160_isrForCrosstalkSources_1031710_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/1b637027-1fab-41c8-a203-a0aa9c998160_isrForCrosstalkSources_1031710_38.stderr parsl.dataflow.dflow INFO: Task 1857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9ee957ee1a0414ab726b77dd0915664 with result from task 1857 parsl.dataflow.dflow DEBUG: Task 1857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a21d9596-742d-41fb-b93d-ab4df4cf67de_isrForCrosstalkSources_1031723_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a21d9596-742d-41fb-b93d-ab4df4cf67de_isrForCrosstalkSources_1031723_2.stderr 2024-03-17 04:06:43 proc_lsst.multi:170 [INFO] [multi] job_id 17148156 proc_lsst.multi INFO: [multi] job_id 17148156 2024-03-17 04:06:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 155 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 155 2024-03-17 04:06:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148156 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148156 parsl.executors.status_handling DEBUG: Launched block 210 on executor multi with job ID 17148156 parsl.executors.status_handling INFO: Allocated block ID 211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-211 2024-03-17 04:06:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 2024-03-17 04:06:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 155 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 155 2024-03-17 04:06:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ae9486d0ee019234e89e9b7da469175 with result from task 1842 parsl.dataflow.dflow DEBUG: Task 1842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/92f9616f-edfc-4793-a1be-840a089e9ae8_isrForCrosstalkSources_1031703_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/92f9616f-edfc-4793-a1be-840a089e9ae8_isrForCrosstalkSources_1031703_18.stderr parsl.dataflow.dflow INFO: Task 1355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e230de0903a19996ee70edaa1e82237b with result from task 1355 parsl.dataflow.dflow DEBUG: Task 1355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a7f57bdb-85b9-4799-adf7-8437db88344f_isrForCrosstalkSources_1031728_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/a7f57bdb-85b9-4799-adf7-8437db88344f_isrForCrosstalkSources_1031728_45.stderr parsl.dataflow.dflow INFO: Task 1224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 647ef3cb581e9c287a8135247918733e with result from task 1224 parsl.dataflow.dflow DEBUG: Task 1224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/7f203e98-715a-4ed6-b61b-1082d7ab18ce_isrForCrosstalkSources_1031728_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/7f203e98-715a-4ed6-b61b-1082d7ab18ce_isrForCrosstalkSources_1031728_11.stderr 2024-03-17 04:06:44 proc_lsst.multi:170 [INFO] [multi] job_id 17148157 proc_lsst.multi INFO: [multi] job_id 17148157 2024-03-17 04:06:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 156 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 156 2024-03-17 04:06:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148157 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148157 parsl.executors.status_handling DEBUG: Launched block 211 on executor multi with job ID 17148157 parsl.executors.status_handling INFO: Allocated block ID 212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-212 2024-03-17 04:06:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 2024-03-17 04:06:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 156 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 156 2024-03-17 04:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8843eef4a5fe576c3971010b3281baf5 with result from task 1640 parsl.dataflow.dflow DEBUG: Task 1640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f37e385b-7fe8-4ea3-93f8-0835cf5d1f8a_isrForCrosstalkSources_1031730_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f37e385b-7fe8-4ea3-93f8-0835cf5d1f8a_isrForCrosstalkSources_1031730_16.stderr parsl.dataflow.dflow INFO: Task 1674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 517fa78dacead25303a11a5c9740f9ec with result from task 1674 parsl.dataflow.dflow DEBUG: Task 1674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/2495545e-202f-4ea6-8cfe-83081777a04a_isrForCrosstalkSources_1031701_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/2495545e-202f-4ea6-8cfe-83081777a04a_isrForCrosstalkSources_1031701_34.stderr parsl.dataflow.dflow INFO: Task 1811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ec43b4914323e1324e16a66901638c4 with result from task 1811 parsl.dataflow.dflow DEBUG: Task 1811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/613043fb-f41a-4488-8894-413fc49fd2fa_isrForCrosstalkSources_1031712_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/613043fb-f41a-4488-8894-413fc49fd2fa_isrForCrosstalkSources_1031712_55.stderr parsl.dataflow.dflow INFO: Task 1826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91d506e7598509690c6839fc7d0d345a with result from task 1826 parsl.dataflow.dflow DEBUG: Task 1826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/5cf090bf-5b37-443f-802a-13e87b78b567_isrForCrosstalkSources_1031708_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/5cf090bf-5b37-443f-802a-13e87b78b567_isrForCrosstalkSources_1031708_33.stderr parsl.dataflow.dflow INFO: Task 1312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50d89f388b816210d8ada114fd379621 with result from task 1312 parsl.dataflow.dflow DEBUG: Task 1312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/21c65279-b20a-4d9a-a14b-09ccb39643cf_isrForCrosstalkSources_1031726_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/21c65279-b20a-4d9a-a14b-09ccb39643cf_isrForCrosstalkSources_1031726_10.stderr parsl.dataflow.dflow INFO: Task 1745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2534862fb6448002c653b0cb818eb77a with result from task 1745 parsl.dataflow.dflow DEBUG: Task 1745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b3695ce4-a64a-44fa-b1db-18abb9cf8e5d_isrForCrosstalkSources_1031712_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b3695ce4-a64a-44fa-b1db-18abb9cf8e5d_isrForCrosstalkSources_1031712_22.stderr parsl.dataflow.dflow INFO: Task 1702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c177e24cb94b8ee95ead5c089b28022 with result from task 1702 parsl.dataflow.dflow DEBUG: Task 1702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4f594657-e528-4095-b685-3c22e031b10b_isrForCrosstalkSources_1031719_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4f594657-e528-4095-b685-3c22e031b10b_isrForCrosstalkSources_1031719_56.stderr parsl.dataflow.dflow INFO: Task 1800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a85d6c41097ea5f70c0e35260fc02500 with result from task 1800 parsl.dataflow.dflow DEBUG: Task 1800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/1065b663-cff9-4af3-b4f4-5944827e5f93_isrForCrosstalkSources_1031704_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/1065b663-cff9-4af3-b4f4-5944827e5f93_isrForCrosstalkSources_1031704_42.stderr parsl.dataflow.dflow INFO: Task 1359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e135730af17e83dddaf3ce2d58ad8453 with result from task 1359 parsl.dataflow.dflow DEBUG: Task 1359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/9554ebb9-0adb-4cde-98fb-3831892eaa3a_isrForCrosstalkSources_1031721_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/9554ebb9-0adb-4cde-98fb-3831892eaa3a_isrForCrosstalkSources_1031721_23.stderr parsl.dataflow.dflow INFO: Task 1932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 685796a5dbb47ca81cf855d37123c404 with result from task 1932 parsl.dataflow.dflow DEBUG: Task 1932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/40f78ae6-e398-48ce-93ee-ca22bad91280_isrForCrosstalkSources_1031718_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/40f78ae6-e398-48ce-93ee-ca22bad91280_isrForCrosstalkSources_1031718_48.stderr 2024-03-17 04:06:44 proc_lsst.multi:170 [INFO] [multi] job_id 17148158 proc_lsst.multi INFO: [multi] job_id 17148158 2024-03-17 04:06:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 157 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 157 2024-03-17 04:06:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148158 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148158 parsl.executors.status_handling DEBUG: Launched block 212 on executor multi with job ID 17148158 parsl.executors.status_handling INFO: Allocated block ID 213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-213 2024-03-17 04:06:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 2024-03-17 04:06:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 157 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 157 2024-03-17 04:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09ef9c89a8c22c121a7e8cfc84eabfc2 with result from task 1234 parsl.dataflow.dflow DEBUG: Task 1234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/9cc89d63-6fa5-4f6a-88c7-5cddf3f7743e_isrForCrosstalkSources_1031723_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/9cc89d63-6fa5-4f6a-88c7-5cddf3f7743e_isrForCrosstalkSources_1031723_1.stderr parsl.dataflow.dflow INFO: Task 1746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 221718351d8007486132a0feb9c20f1d with result from task 1746 parsl.dataflow.dflow DEBUG: Task 1746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/869c2b7e-bc66-4b80-b13b-15d143cbfa39_isrForCrosstalkSources_1031732_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/869c2b7e-bc66-4b80-b13b-15d143cbfa39_isrForCrosstalkSources_1031732_40.stderr parsl.dataflow.dflow INFO: Task 2090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5741087635a3661929c8613f039cbf71 with result from task 2090 parsl.dataflow.dflow DEBUG: Task 2090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/be0eb3e7-dc31-4629-85e8-0432a38dc9b0_isrForCrosstalkSources_1031713_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/be0eb3e7-dc31-4629-85e8-0432a38dc9b0_isrForCrosstalkSources_1031713_10.stderr parsl.dataflow.dflow INFO: Task 1347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa3c9cb06b2a0779a092478c43a91095 with result from task 1347 parsl.dataflow.dflow DEBUG: Task 1347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/8b04162c-9116-4665-803e-dc662060fdcd_isrForCrosstalkSources_1031731_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/8b04162c-9116-4665-803e-dc662060fdcd_isrForCrosstalkSources_1031731_23.stderr 2024-03-17 04:06:45 proc_lsst.multi:170 [INFO] [multi] job_id 17148159 proc_lsst.multi INFO: [multi] job_id 17148159 2024-03-17 04:06:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 158 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 158 2024-03-17 04:06:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148159 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148159 parsl.executors.status_handling DEBUG: Launched block 213 on executor multi with job ID 17148159 parsl.executors.status_handling INFO: Allocated block ID 214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-214 2024-03-17 04:06:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 2024-03-17 04:06:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 158 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 158 2024-03-17 04:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a569db072f39d133d14f21bbfdcf37ac with result from task 1871 parsl.dataflow.dflow DEBUG: Task 1871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e22a1199-38a9-4bb7-b50c-87e98d109a2b_isrForCrosstalkSources_1031706_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e22a1199-38a9-4bb7-b50c-87e98d109a2b_isrForCrosstalkSources_1031706_26.stderr parsl.dataflow.dflow INFO: Task 2091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d89b8ba6d9206fddbbd053359fb75ef with result from task 2091 parsl.dataflow.dflow DEBUG: Task 2091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ccd19f9f-3bc4-4c13-8a57-2628a227ba5f_isrForCrosstalkSources_1031724_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/ccd19f9f-3bc4-4c13-8a57-2628a227ba5f_isrForCrosstalkSources_1031724_59.stderr parsl.dataflow.dflow INFO: Task 2122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f58fd129526fd3f6443b312b9b5cfe1 with result from task 2122 parsl.dataflow.dflow DEBUG: Task 2122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ad5baa43-af15-4da0-8233-12d9e71dee7c_isrForCrosstalkSources_1031713_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ad5baa43-af15-4da0-8233-12d9e71dee7c_isrForCrosstalkSources_1031713_42.stderr parsl.dataflow.dflow INFO: Task 1874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9083942b5c300a379066a3a12f0a57f with result from task 1874 parsl.dataflow.dflow DEBUG: Task 1874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/4c5e4d66-7bfc-4904-9de8-913ba22ccfff_isrForCrosstalkSources_1031707_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/4c5e4d66-7bfc-4904-9de8-913ba22ccfff_isrForCrosstalkSources_1031707_21.stderr parsl.dataflow.dflow INFO: Task 2135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c91f719df6a5cf638c2b13a622b7a6b with result from task 2135 parsl.dataflow.dflow DEBUG: Task 2135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/b1d8d952-0121-4c60-8656-55062eae18a4_isrForCrosstalkSources_1031738_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/b1d8d952-0121-4c60-8656-55062eae18a4_isrForCrosstalkSources_1031738_1.stderr parsl.dataflow.dflow INFO: Task 1305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45740c40360a36d0c46caef6453bfbdc with result from task 1305 parsl.dataflow.dflow DEBUG: Task 1305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8b46b0a8-37bf-4ec4-9a68-a8b8cfdc9a45_isrForCrosstalkSources_1031719_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8b46b0a8-37bf-4ec4-9a68-a8b8cfdc9a45_isrForCrosstalkSources_1031719_29.stderr parsl.dataflow.dflow INFO: Task 1839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 630cf228b5595cdda751f8b5aa8c05ce with result from task 1839 parsl.dataflow.dflow DEBUG: Task 1839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/56bf2726-41bf-4853-99c4-59bf5ff734ec_isrForCrosstalkSources_1031705_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/56bf2726-41bf-4853-99c4-59bf5ff734ec_isrForCrosstalkSources_1031705_32.stderr parsl.dataflow.dflow INFO: Task 2108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1c62777449262bf1d92c262ea4ff47b with result from task 2108 parsl.dataflow.dflow DEBUG: Task 2108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d42b19ae-37a3-46f9-b80a-465b39198371_isrForCrosstalkSources_1031733_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/d42b19ae-37a3-46f9-b80a-465b39198371_isrForCrosstalkSources_1031733_49.stderr parsl.dataflow.dflow INFO: Task 1845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fe9c60560766b62ba661e40c681489b with result from task 1845 parsl.dataflow.dflow DEBUG: Task 1845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/02ed4ea2-f437-4269-8e9d-eba833f9e469_isrForCrosstalkSources_1031719_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/02ed4ea2-f437-4269-8e9d-eba833f9e469_isrForCrosstalkSources_1031719_37.stderr parsl.dataflow.dflow INFO: Task 1912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea1c6dc5cc74d0bca51c302ab9228ac1 with result from task 1912 parsl.dataflow.dflow DEBUG: Task 1912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/5ca89d51-8543-4030-a45f-789c4aac7710_isrForCrosstalkSources_1031706_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/5ca89d51-8543-4030-a45f-789c4aac7710_isrForCrosstalkSources_1031706_17.stderr parsl.dataflow.dflow INFO: Task 2089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee92385d77cd433f5ee15c3b6d1eef2f with result from task 2089 parsl.dataflow.dflow DEBUG: Task 2089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/fd22232f-4b7b-4e2f-add7-2c2106f5496a_isrForCrosstalkSources_1031722_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/fd22232f-4b7b-4e2f-add7-2c2106f5496a_isrForCrosstalkSources_1031722_23.stderr parsl.dataflow.dflow INFO: Task 1884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad2556c3c39efe37d1d3efb2d0ba94c2 with result from task 1884 parsl.dataflow.dflow DEBUG: Task 1884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/c7b1c589-39e4-4c6d-8cab-2a1c4812b4c0_isrForCrosstalkSources_1031701_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/c7b1c589-39e4-4c6d-8cab-2a1c4812b4c0_isrForCrosstalkSources_1031701_32.stderr parsl.dataflow.dflow INFO: Task 2086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cb78cd77ee26dcbd7510bb10a7d2852 with result from task 2086 parsl.dataflow.dflow DEBUG: Task 2086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/d0fdf29e-38d0-49fa-8d69-e5fc42f90b1b_isrForCrosstalkSources_1031702_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/d0fdf29e-38d0-49fa-8d69-e5fc42f90b1b_isrForCrosstalkSources_1031702_42.stderr parsl.dataflow.dflow INFO: Task 1228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee4d674b4fd0edaa897d1787df9f157a with result from task 1228 parsl.dataflow.dflow DEBUG: Task 1228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/fa3de3fd-f8a7-4061-82ce-8b9e45cbac5d_isrForCrosstalkSources_1031714_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/fa3de3fd-f8a7-4061-82ce-8b9e45cbac5d_isrForCrosstalkSources_1031714_43.stderr 2024-03-17 04:06:46 proc_lsst.multi:170 [INFO] [multi] job_id 17148160 proc_lsst.multi INFO: [multi] job_id 17148160 2024-03-17 04:06:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 159 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 159 2024-03-17 04:06:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148160 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148160 parsl.executors.status_handling DEBUG: Launched block 214 on executor multi with job ID 17148160 parsl.executors.status_handling INFO: Allocated block ID 215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-215 2024-03-17 04:06:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 2024-03-17 04:06:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 159 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 159 2024-03-17 04:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 975d4f74570c17161ea9590e905b041f with result from task 1690 parsl.dataflow.dflow DEBUG: Task 1690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/999cc33d-0315-4c60-879f-b8a4af311278_isrForCrosstalkSources_1031699_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/999cc33d-0315-4c60-879f-b8a4af311278_isrForCrosstalkSources_1031699_25.stderr parsl.dataflow.dflow INFO: Task 2181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4308ec69a70e7343c0e9b53d4db9d4a with result from task 2181 parsl.dataflow.dflow DEBUG: Task 2181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/00968709-f2f2-4e9f-b7f3-e5d55db8a772_isrForCrosstalkSources_1031714_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/00968709-f2f2-4e9f-b7f3-e5d55db8a772_isrForCrosstalkSources_1031714_13.stderr parsl.dataflow.dflow INFO: Task 1309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba7ac023010c393e19c4c423321c3829 with result from task 1309 parsl.dataflow.dflow DEBUG: Task 1309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/dedab0be-b843-4cbc-abfb-c4374b31ca7f_isrForCrosstalkSources_1031700_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/dedab0be-b843-4cbc-abfb-c4374b31ca7f_isrForCrosstalkSources_1031700_1.stderr parsl.dataflow.dflow INFO: Task 1242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210a29956f5b5c9988e86388d25a4d3b with result from task 1242 parsl.dataflow.dflow DEBUG: Task 1242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/1d057f3c-aebe-44e1-865b-410bc77028ba_isrForCrosstalkSources_1031699_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/1d057f3c-aebe-44e1-865b-410bc77028ba_isrForCrosstalkSources_1031699_28.stderr parsl.dataflow.dflow INFO: Task 1722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd607ce36ef3246f51d98768a7edfe9f with result from task 1722 parsl.dataflow.dflow DEBUG: Task 1722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4968f366-573d-40f4-aa2c-c1e51e3b0e4f_isrForCrosstalkSources_1031738_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/4968f366-573d-40f4-aa2c-c1e51e3b0e4f_isrForCrosstalkSources_1031738_49.stderr parsl.dataflow.dflow INFO: Task 1206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 087abffa40fb455f8b698c951a557934 with result from task 1206 parsl.dataflow.dflow DEBUG: Task 1206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8fa2f3c4-8429-49f7-922c-325c92633f7f_isrForCrosstalkSources_1031703_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8fa2f3c4-8429-49f7-922c-325c92633f7f_isrForCrosstalkSources_1031703_48.stderr 2024-03-17 04:06:46 proc_lsst.multi:170 [INFO] [multi] job_id 17148161 proc_lsst.multi INFO: [multi] job_id 17148161 2024-03-17 04:06:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 160 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 160 2024-03-17 04:06:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148161 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148161 parsl.executors.status_handling DEBUG: Launched block 215 on executor multi with job ID 17148161 parsl.executors.status_handling INFO: Allocated block ID 216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-216 2024-03-17 04:06:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 2024-03-17 04:06:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 160 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 160 2024-03-17 04:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa9006a2f226596781eabf22a072d221 with result from task 1704 parsl.dataflow.dflow DEBUG: Task 1704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7b946bdc-9a4b-4f8b-8df1-65629936d958_isrForCrosstalkSources_1031729_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/7b946bdc-9a4b-4f8b-8df1-65629936d958_isrForCrosstalkSources_1031729_22.stderr parsl.dataflow.dflow INFO: Task 1910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c566934ccf6d9eb85c2804fb45858a7 with result from task 1910 parsl.dataflow.dflow DEBUG: Task 1910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3d3bf068-abdd-4999-81c5-1d9373ce4bc0_isrForCrosstalkSources_1031719_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3d3bf068-abdd-4999-81c5-1d9373ce4bc0_isrForCrosstalkSources_1031719_2.stderr parsl.dataflow.dflow INFO: Task 1664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4a0ca3aa43f0654d096b86c3935f2b5 with result from task 1664 parsl.dataflow.dflow DEBUG: Task 1664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/75221928-b092-4d6b-9a3f-ad132fa289e1_isrForCrosstalkSources_1031713_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/75221928-b092-4d6b-9a3f-ad132fa289e1_isrForCrosstalkSources_1031713_32.stderr parsl.dataflow.dflow INFO: Task 1304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 212e5b7cb3bd46c93ebcc5b5c28c117c with result from task 1304 parsl.dataflow.dflow DEBUG: Task 1304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/1db45182-d5d4-4baa-aa1a-b7be52c7d330_isrForCrosstalkSources_1031730_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/1db45182-d5d4-4baa-aa1a-b7be52c7d330_isrForCrosstalkSources_1031730_7.stderr parsl.dataflow.dflow INFO: Task 1734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 057d1da526febf34b56b01db98eedfd5 with result from task 1734 parsl.dataflow.dflow DEBUG: Task 1734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/944dab53-6d4a-4abf-bea6-ee6a0606c017_isrForCrosstalkSources_1031706_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/944dab53-6d4a-4abf-bea6-ee6a0606c017_isrForCrosstalkSources_1031706_18.stderr parsl.dataflow.dflow INFO: Task 1762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79b34f74d39b385e2eca65b42770e9ad with result from task 1762 parsl.dataflow.dflow DEBUG: Task 1762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/430ae65c-6f69-48af-9a01-0e004867110c_isrForCrosstalkSources_1031702_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/430ae65c-6f69-48af-9a01-0e004867110c_isrForCrosstalkSources_1031702_37.stderr parsl.dataflow.dflow INFO: Task 1761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 640f9c796a79bf44c93baf0bf762dbca with result from task 1761 parsl.dataflow.dflow DEBUG: Task 1761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0efa7d15-4300-4f88-8243-fccd9d201a4d_isrForCrosstalkSources_1031706_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0efa7d15-4300-4f88-8243-fccd9d201a4d_isrForCrosstalkSources_1031706_31.stderr parsl.dataflow.dflow INFO: Task 1889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 759bdb2ae83d97e3fb83f6932ecd56a7 with result from task 1889 parsl.dataflow.dflow DEBUG: Task 1889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f5c8ac6f-d47a-46e1-b451-c484198fa128_isrForCrosstalkSources_1031732_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/f5c8ac6f-d47a-46e1-b451-c484198fa128_isrForCrosstalkSources_1031732_23.stderr parsl.dataflow.dflow INFO: Task 1795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fe2aa0763bf6dcf068ade99023e7328 with result from task 1795 parsl.dataflow.dflow DEBUG: Task 1795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a508ae91-b330-4e6e-a1de-45980d826378_isrForCrosstalkSources_1031712_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/a508ae91-b330-4e6e-a1de-45980d826378_isrForCrosstalkSources_1031712_51.stderr 2024-03-17 04:06:47 proc_lsst.multi:170 [INFO] [multi] job_id 17148162 proc_lsst.multi INFO: [multi] job_id 17148162 2024-03-17 04:06:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 161 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 161 2024-03-17 04:06:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148162 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148162 parsl.executors.status_handling DEBUG: Launched block 216 on executor multi with job ID 17148162 parsl.executors.status_handling INFO: Allocated block ID 217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-217 2024-03-17 04:06:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 2024-03-17 04:06:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 161 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 161 2024-03-17 04:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfe94c97527d29fd80759e1239660372 with result from task 1787 parsl.dataflow.dflow DEBUG: Task 1787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d3d3a80c-37eb-4cc3-8d09-b2faebf6509b_isrForCrosstalkSources_1031714_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d3d3a80c-37eb-4cc3-8d09-b2faebf6509b_isrForCrosstalkSources_1031714_28.stderr parsl.dataflow.dflow INFO: Task 1308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 811ea236b91678c942a84cb0c4fa7a90 with result from task 1308 parsl.dataflow.dflow DEBUG: Task 1308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a70254c0-12cc-4840-8231-d7b46efbce68_isrForCrosstalkSources_1031725_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a70254c0-12cc-4840-8231-d7b46efbce68_isrForCrosstalkSources_1031725_30.stderr parsl.dataflow.dflow INFO: Task 1849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 974f24606237aa7d4a052aefbdfdc4b4 with result from task 1849 parsl.dataflow.dflow DEBUG: Task 1849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/cd2405a2-4103-4473-8645-9221761cb3fb_isrForCrosstalkSources_1031732_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/cd2405a2-4103-4473-8645-9221761cb3fb_isrForCrosstalkSources_1031732_51.stderr parsl.dataflow.dflow INFO: Task 1721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c850cbc5bc7cce5bdcc62c1a298d6a6e with result from task 1721 parsl.dataflow.dflow DEBUG: Task 1721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a80ba9f7-da70-448a-9053-cae7a836517e_isrForCrosstalkSources_1031736_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/a80ba9f7-da70-448a-9053-cae7a836517e_isrForCrosstalkSources_1031736_18.stderr parsl.dataflow.dflow INFO: Task 2189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1576da178e51ac6ac8665f42dd8270db with result from task 2189 parsl.dataflow.dflow DEBUG: Task 2189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/41590ab8-6ad9-4257-9ff5-701af7c309ad_isrForCrosstalkSources_1031725_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/41590ab8-6ad9-4257-9ff5-701af7c309ad_isrForCrosstalkSources_1031725_39.stderr parsl.dataflow.dflow INFO: Task 1607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a60bf4d21e97ecb588972dcd38c891d9 with result from task 1607 parsl.dataflow.dflow DEBUG: Task 1607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/debb2906-8456-4b3c-8277-2ae044a53a56_isrForCrosstalkSources_1031706_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/debb2906-8456-4b3c-8277-2ae044a53a56_isrForCrosstalkSources_1031706_54.stderr parsl.dataflow.dflow INFO: Task 1668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e582013edc7ba53b24ba5b99db91bd99 with result from task 1668 parsl.dataflow.dflow DEBUG: Task 1668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d1379b56-77cf-48ed-9b4a-da764feaadd4_isrForCrosstalkSources_1031726_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d1379b56-77cf-48ed-9b4a-da764feaadd4_isrForCrosstalkSources_1031726_13.stderr parsl.dataflow.dflow INFO: Task 2185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d949c5ac197fda9ce9c65d726ef53f60 with result from task 2185 parsl.dataflow.dflow DEBUG: Task 2185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/8cd39082-900d-4848-ac26-caa99530fa0d_isrForCrosstalkSources_1031737_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/8cd39082-900d-4848-ac26-caa99530fa0d_isrForCrosstalkSources_1031737_47.stderr parsl.dataflow.dflow INFO: Task 1836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41ebe45ddc82306e36c76f1da6fd0f76 with result from task 1836 parsl.dataflow.dflow DEBUG: Task 1836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1972e91d-0c3a-4de5-95f9-3b5f36be4864_isrForCrosstalkSources_1031703_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1972e91d-0c3a-4de5-95f9-3b5f36be4864_isrForCrosstalkSources_1031703_1.stderr parsl.dataflow.dflow INFO: Task 1341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4554306e701768ecd6a1f1c50e7aadb with result from task 1341 parsl.dataflow.dflow DEBUG: Task 1341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/a583b80d-748e-4bea-b766-a22d8a919196_isrForCrosstalkSources_1031717_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/a583b80d-748e-4bea-b766-a22d8a919196_isrForCrosstalkSources_1031717_3.stderr parsl.dataflow.dflow INFO: Task 1688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c90bd9c5286514cd9cf98566db317746 with result from task 1688 parsl.dataflow.dflow DEBUG: Task 1688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/9943b647-8275-4f2f-b491-1c2c26009bfd_isrForCrosstalkSources_1031732_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/9943b647-8275-4f2f-b491-1c2c26009bfd_isrForCrosstalkSources_1031732_33.stderr parsl.dataflow.dflow INFO: Task 1793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43e5b57ec6b2908d347144f2a41c00d0 with result from task 1793 parsl.dataflow.dflow DEBUG: Task 1793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ccba5059-d92d-4bd6-a230-c37ce6bd5811_isrForCrosstalkSources_1031731_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ccba5059-d92d-4bd6-a230-c37ce6bd5811_isrForCrosstalkSources_1031731_24.stderr 2024-03-17 04:06:47 proc_lsst.multi:170 [INFO] [multi] job_id 17148163 proc_lsst.multi INFO: [multi] job_id 17148163 2024-03-17 04:06:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 162 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 162 2024-03-17 04:06:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148163 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148163 parsl.executors.status_handling DEBUG: Launched block 217 on executor multi with job ID 17148163 parsl.executors.status_handling INFO: Allocated block ID 218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-218 2024-03-17 04:06:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 2024-03-17 04:06:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 162 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 162 2024-03-17 04:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 741927ebfbd9c31ab79e6ed780212740 with result from task 1356 parsl.dataflow.dflow DEBUG: Task 1356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/c6656c0a-4e67-4361-8f18-178a88b9160d_isrForCrosstalkSources_1031717_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/c6656c0a-4e67-4361-8f18-178a88b9160d_isrForCrosstalkSources_1031717_17.stderr parsl.dataflow.dflow INFO: Task 2184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4dbfe02a18f343a2f95e7b238c0852f with result from task 2184 parsl.dataflow.dflow DEBUG: Task 2184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0ba34f58-9c07-4d8f-a8d7-48d0db714ff5_isrForCrosstalkSources_1031732_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0ba34f58-9c07-4d8f-a8d7-48d0db714ff5_isrForCrosstalkSources_1031732_62.stderr parsl.dataflow.dflow INFO: Task 2179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40bbb7e54b43cc5cbb5fcf141ac59109 with result from task 2179 parsl.dataflow.dflow DEBUG: Task 2179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/2d76a4c6-d5d5-4f52-9d70-958b8ec1c1c7_isrForCrosstalkSources_1031720_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/2d76a4c6-d5d5-4f52-9d70-958b8ec1c1c7_isrForCrosstalkSources_1031720_51.stderr parsl.dataflow.dflow INFO: Task 1772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81cff3b4a98f00fee29be2f9a9b52f6f with result from task 1772 parsl.dataflow.dflow DEBUG: Task 1772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/550e8cb9-3738-48dd-8016-4886700bcfb3_isrForCrosstalkSources_1031701_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/550e8cb9-3738-48dd-8016-4886700bcfb3_isrForCrosstalkSources_1031701_3.stderr parsl.dataflow.dflow INFO: Task 1948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 137aed51bdefe056d1c62fbe3ac9dd5f with result from task 1948 parsl.dataflow.dflow DEBUG: Task 1948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5bbdaefd-4636-46bd-883f-2afe2a6de33a_isrForCrosstalkSources_1031701_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/5bbdaefd-4636-46bd-883f-2afe2a6de33a_isrForCrosstalkSources_1031701_18.stderr parsl.dataflow.dflow INFO: Task 1766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fc84ac8fb1c622610450cf903d4fa45 with result from task 1766 parsl.dataflow.dflow DEBUG: Task 1766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5f7fc1d8-5bd0-4bc9-8dd2-c16bea5d9840_isrForCrosstalkSources_1031724_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5f7fc1d8-5bd0-4bc9-8dd2-c16bea5d9840_isrForCrosstalkSources_1031724_54.stderr parsl.dataflow.dflow INFO: Task 1682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c57dd6ec6b0a142a7cf36aae1005924 with result from task 1682 parsl.dataflow.dflow DEBUG: Task 1682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6e5c6fc2-6932-4a59-b0cf-74b2e3b762af_isrForCrosstalkSources_1031733_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6e5c6fc2-6932-4a59-b0cf-74b2e3b762af_isrForCrosstalkSources_1031733_13.stderr parsl.dataflow.dflow INFO: Task 1611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18f7e9f460d7773350784a3a2707908e with result from task 1611 parsl.dataflow.dflow DEBUG: Task 1611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8ea9c130-b6f0-4408-938c-eddc9f13b420_isrForCrosstalkSources_1031729_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/8ea9c130-b6f0-4408-938c-eddc9f13b420_isrForCrosstalkSources_1031729_16.stderr parsl.dataflow.dflow INFO: Task 1786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65aee4ecc4bd84ea24bb66f541ef2a11 with result from task 1786 parsl.dataflow.dflow DEBUG: Task 1786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/3c0f7323-24ee-4c7c-b815-a858fbcd72f6_isrForCrosstalkSources_1031703_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/3c0f7323-24ee-4c7c-b815-a858fbcd72f6_isrForCrosstalkSources_1031703_26.stderr parsl.dataflow.dflow INFO: Task 1815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 810d99a9166dae1617f723f914de844e with result from task 1815 parsl.dataflow.dflow DEBUG: Task 1815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/21c6ee5f-e5c8-49ed-b7dc-becfc78e87e5_isrForCrosstalkSources_1031732_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/21c6ee5f-e5c8-49ed-b7dc-becfc78e87e5_isrForCrosstalkSources_1031732_52.stderr parsl.dataflow.dflow INFO: Task 1210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dd8b2864d61e580904de97f7d6bf885 with result from task 1210 parsl.dataflow.dflow DEBUG: Task 1210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a251471b-7369-4b06-83ca-88134f081710_isrForCrosstalkSources_1031718_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a251471b-7369-4b06-83ca-88134f081710_isrForCrosstalkSources_1031718_43.stderr parsl.dataflow.dflow INFO: Task 2186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e87b09e63edf063b28dce971758b39f1 with result from task 2186 parsl.dataflow.dflow DEBUG: Task 2186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/02b4a96d-e856-4012-ba28-9c1996a81be0_isrForCrosstalkSources_1031708_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/02b4a96d-e856-4012-ba28-9c1996a81be0_isrForCrosstalkSources_1031708_31.stderr parsl.dataflow.dflow INFO: Task 1398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b656489d7bb4f9750d0648a458806e0b with result from task 1398 parsl.dataflow.dflow DEBUG: Task 1398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c47d3aed-c5b0-49eb-9408-4e0de761a251_isrForCrosstalkSources_1031705_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c47d3aed-c5b0-49eb-9408-4e0de761a251_isrForCrosstalkSources_1031705_27.stderr parsl.dataflow.dflow INFO: Task 1375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32451126a56f4f776fe5cf6be89dbf9b with result from task 1375 parsl.dataflow.dflow DEBUG: Task 1375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/d383a7d2-1006-4631-8a3b-dcaeea907fbe_isrForCrosstalkSources_1031717_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/d383a7d2-1006-4631-8a3b-dcaeea907fbe_isrForCrosstalkSources_1031717_19.stderr parsl.dataflow.dflow INFO: Task 1624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb7d74cad41a999631430172a745500a with result from task 1624 parsl.dataflow.dflow DEBUG: Task 1624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1029e10a-dd08-40f6-a1af-201a6483ecfa_isrForCrosstalkSources_1031732_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/1029e10a-dd08-40f6-a1af-201a6483ecfa_isrForCrosstalkSources_1031732_4.stderr parsl.dataflow.dflow INFO: Task 1396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 523ae5645fad7da5585732ed36eefd39 with result from task 1396 parsl.dataflow.dflow DEBUG: Task 1396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/7ced527e-84e0-44f9-b592-e115a6a629ed_isrForCrosstalkSources_1031699_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/7ced527e-84e0-44f9-b592-e115a6a629ed_isrForCrosstalkSources_1031699_33.stderr parsl.dataflow.dflow INFO: Task 1395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f40ec3d20c4d63d04567f593753ed00c with result from task 1395 parsl.dataflow.dflow DEBUG: Task 1395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/dc88096e-cef6-49f4-b242-41f1f159a1f5_isrForCrosstalkSources_1031716_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/dc88096e-cef6-49f4-b242-41f1f159a1f5_isrForCrosstalkSources_1031716_34.stderr parsl.dataflow.dflow INFO: Task 1392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef073fe000eaa133871b1120649f3cce with result from task 1392 parsl.dataflow.dflow DEBUG: Task 1392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/41cb5dfa-4abe-4b19-8f72-e81f8542d8ee_isrForCrosstalkSources_1031727_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/41cb5dfa-4abe-4b19-8f72-e81f8542d8ee_isrForCrosstalkSources_1031727_46.stderr parsl.dataflow.dflow INFO: Task 1377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9e9c3d1645e97ff53b84cb4b600898c with result from task 1377 parsl.dataflow.dflow DEBUG: Task 1377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f2196306-2011-43f9-93cb-f310c9067180_isrForCrosstalkSources_1031703_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f2196306-2011-43f9-93cb-f310c9067180_isrForCrosstalkSources_1031703_32.stderr parsl.dataflow.dflow INFO: Task 1374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a17a35ec92e15b369eccbcf528cead2a with result from task 1374 parsl.dataflow.dflow DEBUG: Task 1374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/da671006-1cf5-4250-8a39-3b446b81fec4_isrForCrosstalkSources_1031729_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/da671006-1cf5-4250-8a39-3b446b81fec4_isrForCrosstalkSources_1031729_20.stderr parsl.dataflow.dflow INFO: Task 1803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 938b5d6bf3f94b163485e949346b92df with result from task 1803 parsl.dataflow.dflow DEBUG: Task 1803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f93adf99-fa18-433c-b6fc-f1abe0e9fd1a_isrForCrosstalkSources_1031710_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/f93adf99-fa18-433c-b6fc-f1abe0e9fd1a_isrForCrosstalkSources_1031710_58.stderr parsl.dataflow.dflow INFO: Task 1689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aae8d93a6a30f0f102566921d3d6d5d2 with result from task 1689 parsl.dataflow.dflow DEBUG: Task 1689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7d312f4b-a508-4df8-86ab-01823f3985bf_isrForCrosstalkSources_1031701_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7d312f4b-a508-4df8-86ab-01823f3985bf_isrForCrosstalkSources_1031701_40.stderr parsl.dataflow.dflow INFO: Task 1394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07beb7ab5cb0689f2c09d135530ab474 with result from task 1394 parsl.dataflow.dflow DEBUG: Task 1394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/518bcb97-a917-4058-87b7-d48c8f6b6ef5_isrForCrosstalkSources_1031699_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/518bcb97-a917-4058-87b7-d48c8f6b6ef5_isrForCrosstalkSources_1031699_45.stderr parsl.dataflow.dflow INFO: Task 1711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2880cdefe32871cf69cbe24e030636a8 with result from task 1711 parsl.dataflow.dflow DEBUG: Task 1711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/058250f1-5cbe-4f01-aa79-ec32354f9dd9_isrForCrosstalkSources_1031713_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/058250f1-5cbe-4f01-aa79-ec32354f9dd9_isrForCrosstalkSources_1031713_15.stderr parsl.dataflow.dflow INFO: Task 1203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24a97159c6ab835704428ed325bcc3b6 with result from task 1203 parsl.dataflow.dflow DEBUG: Task 1203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/26d0d23c-eaf1-4198-9530-9df50c758276_isrForCrosstalkSources_1031705_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/26d0d23c-eaf1-4198-9530-9df50c758276_isrForCrosstalkSources_1031705_43.stderr parsl.dataflow.dflow INFO: Task 1616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed3ac04393dc611ef376300f29f4ce88 with result from task 1616 parsl.dataflow.dflow DEBUG: Task 1616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/931b58ce-97ad-4670-a694-82ec2d0a80ee_isrForCrosstalkSources_1031735_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/931b58ce-97ad-4670-a694-82ec2d0a80ee_isrForCrosstalkSources_1031735_21.stderr parsl.dataflow.dflow INFO: Task 1678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37f42f8b26314f94975f330d9b15f435 with result from task 1678 parsl.dataflow.dflow DEBUG: Task 1678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/2c60f4c4-5c24-4d28-9850-7cdf6ca36c43_isrForCrosstalkSources_1031705_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/2c60f4c4-5c24-4d28-9850-7cdf6ca36c43_isrForCrosstalkSources_1031705_13.stderr parsl.dataflow.dflow INFO: Task 1990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32be1e2361b2cc81ad57b48ca4b69106 with result from task 1990 parsl.dataflow.dflow DEBUG: Task 1990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/2fc98b34-f312-4f6c-9c16-1cc564a57e81_isrForCrosstalkSources_1031710_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/2fc98b34-f312-4f6c-9c16-1cc564a57e81_isrForCrosstalkSources_1031710_26.stderr parsl.dataflow.dflow INFO: Task 1706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b41d4f04202bb9c480138925b9152c8 with result from task 1706 parsl.dataflow.dflow DEBUG: Task 1706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/95e5005b-f304-4c1d-9185-6141fd4b94c7_isrForCrosstalkSources_1031707_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/95e5005b-f304-4c1d-9185-6141fd4b94c7_isrForCrosstalkSources_1031707_13.stderr parsl.dataflow.dflow INFO: Task 1946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecf0340b491e42cd0c481492ffbc3231 with result from task 1946 parsl.dataflow.dflow DEBUG: Task 1946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/6650b27f-4d16-47e2-a897-15ae52043a06_isrForCrosstalkSources_1031734_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/6650b27f-4d16-47e2-a897-15ae52043a06_isrForCrosstalkSources_1031734_21.stderr parsl.dataflow.dflow INFO: Task 1727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 942b1436b5e95660293f8e0e83e106bf with result from task 1727 parsl.dataflow.dflow DEBUG: Task 1727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6b750332-8b4d-48a6-8c43-40896a4bd0c2_isrForCrosstalkSources_1031733_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/6b750332-8b4d-48a6-8c43-40896a4bd0c2_isrForCrosstalkSources_1031733_24.stderr parsl.dataflow.dflow INFO: Task 1941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acdd642d210ca75a02f0071064a9eeec with result from task 1941 parsl.dataflow.dflow DEBUG: Task 1941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c4e6219f-6f16-4223-81da-f11eddb747cf_isrForCrosstalkSources_1031719_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c4e6219f-6f16-4223-81da-f11eddb747cf_isrForCrosstalkSources_1031719_34.stderr parsl.dataflow.dflow INFO: Task 1785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fea4710704072bedf65c4d205dcdd6b with result from task 1785 parsl.dataflow.dflow DEBUG: Task 1785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/34b96074-4791-4044-83fd-d29b381834b5_isrForCrosstalkSources_1031705_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/34b96074-4791-4044-83fd-d29b381834b5_isrForCrosstalkSources_1031705_9.stderr parsl.dataflow.dflow INFO: Task 1758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe92e477e777e8ce35b818595043edc with result from task 1758 parsl.dataflow.dflow DEBUG: Task 1758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/82898554-4c64-4450-a9f6-6892105733df_isrForCrosstalkSources_1031735_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/82898554-4c64-4450-a9f6-6892105733df_isrForCrosstalkSources_1031735_34.stderr parsl.dataflow.dflow INFO: Task 1310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee2bcf5a33ced60c7597ac6574f50727 with result from task 1310 parsl.dataflow.dflow DEBUG: Task 1310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1e81bb80-71a5-43f5-8a17-4aab894630f3_isrForCrosstalkSources_1031701_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/1e81bb80-71a5-43f5-8a17-4aab894630f3_isrForCrosstalkSources_1031701_60.stderr parsl.dataflow.dflow INFO: Task 2013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da05617141d0e7d768f5c7715b564e99 with result from task 2013 parsl.dataflow.dflow DEBUG: Task 2013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/2ffee46c-215a-48b8-853d-7e290f68945b_isrForCrosstalkSources_1031713_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/2ffee46c-215a-48b8-853d-7e290f68945b_isrForCrosstalkSources_1031713_25.stderr parsl.dataflow.dflow INFO: Task 1724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bea27a8ce444ce4ccbe45d7b5a0c64b1 with result from task 1724 parsl.dataflow.dflow DEBUG: Task 1724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/0491953e-3e62-455f-83d3-1c138d15eb0d_isrForCrosstalkSources_1031699_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/0491953e-3e62-455f-83d3-1c138d15eb0d_isrForCrosstalkSources_1031699_3.stderr parsl.dataflow.dflow INFO: Task 1703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 340f96eb11ee4d698e1e6ec307dcf90e with result from task 1703 parsl.dataflow.dflow DEBUG: Task 1703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/81fd8599-7270-41f1-aacf-9bcc38862623_isrForCrosstalkSources_1031707_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/81fd8599-7270-41f1-aacf-9bcc38862623_isrForCrosstalkSources_1031707_15.stderr parsl.dataflow.dflow INFO: Task 1300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f1961ee4b71047c48ff748ce327025b with result from task 1300 parsl.dataflow.dflow DEBUG: Task 1300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/16ecc51d-7d31-41c9-b49a-2a4268af422e_isrForCrosstalkSources_1031735_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/16ecc51d-7d31-41c9-b49a-2a4268af422e_isrForCrosstalkSources_1031735_55.stderr parsl.dataflow.dflow INFO: Task 1822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8ccd971d5b029e7d626be4a8755d188 with result from task 1822 parsl.dataflow.dflow DEBUG: Task 1822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/96418f74-2420-412d-a439-e3fec4088dd7_isrForCrosstalkSources_1031701_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/96418f74-2420-412d-a439-e3fec4088dd7_isrForCrosstalkSources_1031701_28.stderr parsl.dataflow.dflow INFO: Task 1759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80b59bb47aeb967d54948ab87b568d79 with result from task 1759 parsl.dataflow.dflow DEBUG: Task 1759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e6f83fd0-d39b-440d-b5eb-2a96dbee95e4_isrForCrosstalkSources_1031712_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/e6f83fd0-d39b-440d-b5eb-2a96dbee95e4_isrForCrosstalkSources_1031712_59.stderr parsl.dataflow.dflow INFO: Task 1843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4006945da1d73c025167dd32cfdc92ad with result from task 1843 parsl.dataflow.dflow DEBUG: Task 1843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a8f938b5-7300-427d-97a2-77ef86470395_isrForCrosstalkSources_1031706_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a8f938b5-7300-427d-97a2-77ef86470395_isrForCrosstalkSources_1031706_45.stderr parsl.dataflow.dflow INFO: Task 1726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 106ab3c2c14c6a6e5779bdb0aca4c47a with result from task 1726 parsl.dataflow.dflow DEBUG: Task 1726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/8945d369-5a2c-44d5-af6f-707424a0ef3e_isrForCrosstalkSources_1031711_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/8945d369-5a2c-44d5-af6f-707424a0ef3e_isrForCrosstalkSources_1031711_1.stderr parsl.dataflow.dflow INFO: Task 1230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72ff52aaecb69326cf9e6ba9278d932e with result from task 1230 parsl.dataflow.dflow DEBUG: Task 1230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/89e0e916-480c-4641-a4dc-0ad61b53fd7d_isrForCrosstalkSources_1031713_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/89e0e916-480c-4641-a4dc-0ad61b53fd7d_isrForCrosstalkSources_1031713_41.stderr parsl.dataflow.dflow INFO: Task 1376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b55ca4124e480cfe5868045a9123ad15 with result from task 1376 parsl.dataflow.dflow DEBUG: Task 1376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/12d38d0e-6a43-40bf-8d24-737b734beb83_isrForCrosstalkSources_1031731_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/12d38d0e-6a43-40bf-8d24-737b734beb83_isrForCrosstalkSources_1031731_16.stderr parsl.dataflow.dflow INFO: Task 1243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 477a8f8422fa8e7f3ad7d3bb19bd5ab6 with result from task 1243 parsl.dataflow.dflow DEBUG: Task 1243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bca6dc3e-270c-4174-b07b-f1f1db5643fb_isrForCrosstalkSources_1031723_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bca6dc3e-270c-4174-b07b-f1f1db5643fb_isrForCrosstalkSources_1031723_54.stderr parsl.dataflow.dflow INFO: Task 1749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3a3b1833cdb209a36692a7eeb1af4f1 with result from task 1749 parsl.dataflow.dflow DEBUG: Task 1749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/58693ca1-4ca8-4a7a-a568-094a3137627f_isrForCrosstalkSources_1031718_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/58693ca1-4ca8-4a7a-a568-094a3137627f_isrForCrosstalkSources_1031718_31.stderr parsl.dataflow.dflow INFO: Task 1828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ea1cb1bd34e4d587a45d0e62926379b with result from task 1828 parsl.dataflow.dflow DEBUG: Task 1828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/192f8f1e-42a6-4dee-9f9c-dfeb71ce4802_isrForCrosstalkSources_1031724_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/192f8f1e-42a6-4dee-9f9c-dfeb71ce4802_isrForCrosstalkSources_1031724_33.stderr parsl.dataflow.dflow INFO: Task 1354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce6a3a239698711f7849d5b5b46db687 with result from task 1354 parsl.dataflow.dflow DEBUG: Task 1354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/abad4e7e-b697-4fe1-8feb-dbcae4aa520a_isrForCrosstalkSources_1031720_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/abad4e7e-b697-4fe1-8feb-dbcae4aa520a_isrForCrosstalkSources_1031720_35.stderr parsl.dataflow.dflow INFO: Task 2231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7da8edee20e852101fbc538f919eb4a with result from task 2231 parsl.dataflow.dflow DEBUG: Task 2231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/8811da29-ec3f-418a-84ff-7b4cb153108c_isrForCrosstalkSources_1031713_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/8811da29-ec3f-418a-84ff-7b4cb153108c_isrForCrosstalkSources_1031713_48.stderr parsl.dataflow.dflow INFO: Task 1630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb9d1adfdfbc159309f608bb7d73a3b9 with result from task 1630 parsl.dataflow.dflow DEBUG: Task 1630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/13c82374-8a4a-448b-af9b-866986579058_isrForCrosstalkSources_1031707_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/13c82374-8a4a-448b-af9b-866986579058_isrForCrosstalkSources_1031707_41.stderr parsl.dataflow.dflow INFO: Task 1827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67d1cd601a221ee3b4f50069964b48fa with result from task 1827 parsl.dataflow.dflow DEBUG: Task 1827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/61088146-738e-48f5-8f54-d65681030fb6_isrForCrosstalkSources_1031711_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/61088146-738e-48f5-8f54-d65681030fb6_isrForCrosstalkSources_1031711_47.stderr parsl.dataflow.dflow INFO: Task 1708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 947a1da46d4367839e08fea3d30a85b5 with result from task 1708 parsl.dataflow.dflow DEBUG: Task 1708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/633745d9-b2ea-4d25-ad49-d45fc0c40c99_isrForCrosstalkSources_1031709_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/633745d9-b2ea-4d25-ad49-d45fc0c40c99_isrForCrosstalkSources_1031709_56.stderr 2024-03-17 04:06:50 proc_lsst.multi:170 [INFO] [multi] job_id 17148166 proc_lsst.multi INFO: [multi] job_id 17148166 2024-03-17 04:06:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 163 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 163 2024-03-17 04:06:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148166 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148166 parsl.executors.status_handling DEBUG: Launched block 218 on executor multi with job ID 17148166 parsl.executors.status_handling INFO: Allocated block ID 219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-219 2024-03-17 04:06:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 2024-03-17 04:06:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 163 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 163 2024-03-17 04:06:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7b64ba52e3ca5c91869b9b8e3b4b02f with result from task 1939 parsl.dataflow.dflow DEBUG: Task 1939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/29c83bc4-030b-474f-b4be-946b45ff3e8e_isrForCrosstalkSources_1031725_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/29c83bc4-030b-474f-b4be-946b45ff3e8e_isrForCrosstalkSources_1031725_20.stderr parsl.dataflow.dflow INFO: Task 1741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 356e75378cf6258cd25ac698e919b89a with result from task 1741 parsl.dataflow.dflow DEBUG: Task 1741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e871bfe2-9bfc-4aa8-9ad0-3376a8bf80cc_isrForCrosstalkSources_1031706_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e871bfe2-9bfc-4aa8-9ad0-3376a8bf80cc_isrForCrosstalkSources_1031706_48.stderr parsl.dataflow.dflow INFO: Task 1235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfdde89503da8a3079b7a82eb81cfde4 with result from task 1235 parsl.dataflow.dflow DEBUG: Task 1235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/b0602fc4-e368-480a-a5c7-b9a857bc4f39_isrForCrosstalkSources_1031711_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/b0602fc4-e368-480a-a5c7-b9a857bc4f39_isrForCrosstalkSources_1031711_62.stderr parsl.dataflow.dflow INFO: Task 1208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b333be2b66d10214cc538db78bdd651 with result from task 1208 parsl.dataflow.dflow DEBUG: Task 1208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8445e70a-de9c-4cc7-af0b-b357b4f9c904_isrForCrosstalkSources_1031704_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/8445e70a-de9c-4cc7-af0b-b357b4f9c904_isrForCrosstalkSources_1031704_41.stderr parsl.dataflow.dflow INFO: Task 1824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d0e5c31372c244a0c988f51170c69ec with result from task 1824 parsl.dataflow.dflow DEBUG: Task 1824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c763f48c-6d15-48fc-a4a4-4bd21eed4f5b_isrForCrosstalkSources_1031731_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c763f48c-6d15-48fc-a4a4-4bd21eed4f5b_isrForCrosstalkSources_1031731_51.stderr parsl.dataflow.dflow INFO: Task 2251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b00bb207a9d7a4f2d5db04be217518ee with result from task 2251 parsl.dataflow.dflow DEBUG: Task 2251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/701c56d9-68e2-4509-9b25-52ccd999f4e9_isrForCrosstalkSources_1031708_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/701c56d9-68e2-4509-9b25-52ccd999f4e9_isrForCrosstalkSources_1031708_60.stderr parsl.dataflow.dflow INFO: Task 1760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc63b751ed3a8b5ef64838cd958c94a9 with result from task 1760 parsl.dataflow.dflow DEBUG: Task 1760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5e19234b-6c5b-4efa-891c-1d44970fbea8_isrForCrosstalkSources_1031728_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5e19234b-6c5b-4efa-891c-1d44970fbea8_isrForCrosstalkSources_1031728_7.stderr parsl.dataflow.dflow INFO: Task 1825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58ff9d36d4233a61855ca22b50a0c7f4 with result from task 1825 parsl.dataflow.dflow DEBUG: Task 1825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/1224ea4f-271c-43ab-a17f-65c230d07a0f_isrForCrosstalkSources_1031731_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/1224ea4f-271c-43ab-a17f-65c230d07a0f_isrForCrosstalkSources_1031731_60.stderr parsl.dataflow.dflow INFO: Task 1379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 660b732b888d351e2437884076a7cdc4 with result from task 1379 parsl.dataflow.dflow DEBUG: Task 1379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/833218a8-8fcf-41fd-a7f7-ec4c52d6617f_isrForCrosstalkSources_1031728_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/833218a8-8fcf-41fd-a7f7-ec4c52d6617f_isrForCrosstalkSources_1031728_12.stderr parsl.dataflow.dflow INFO: Task 1225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e6b9ae9543a0fdaa887ca473a3bf9f2 with result from task 1225 parsl.dataflow.dflow DEBUG: Task 1225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/72e8b132-0031-4b07-9e5d-9c4cdb84774b_isrForCrosstalkSources_1031716_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/72e8b132-0031-4b07-9e5d-9c4cdb84774b_isrForCrosstalkSources_1031716_43.stderr parsl.dataflow.dflow INFO: Task 1817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f90c8fc1861a86d10d9877a28df04c7 with result from task 1817 parsl.dataflow.dflow DEBUG: Task 1817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/9b6d913d-86bc-4680-95e8-0d517d768334_isrForCrosstalkSources_1031731_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/9b6d913d-86bc-4680-95e8-0d517d768334_isrForCrosstalkSources_1031731_49.stderr parsl.dataflow.dflow INFO: Task 2234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43a56d813f1a5c024e48f1edd991e1c4 with result from task 2234 parsl.dataflow.dflow DEBUG: Task 2234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4831358a-27de-4255-bfc1-cd7e35e1aa4b_isrForCrosstalkSources_1031719_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/4831358a-27de-4255-bfc1-cd7e35e1aa4b_isrForCrosstalkSources_1031719_18.stderr parsl.dataflow.dflow INFO: Task 1835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bf4299c0274d27ca324a5bb1650686b with result from task 1835 parsl.dataflow.dflow DEBUG: Task 1835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/fa09d9cc-828c-41f8-899b-ce4bbe00278f_isrForCrosstalkSources_1031709_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/fa09d9cc-828c-41f8-899b-ce4bbe00278f_isrForCrosstalkSources_1031709_1.stderr parsl.dataflow.dflow INFO: Task 2274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44a67df91e7bc203914ea9244fd511ae with result from task 2274 parsl.dataflow.dflow DEBUG: Task 2274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a4ac7de7-aa2e-4afb-a843-9b87f2eee58a_isrForCrosstalkSources_1031719_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a4ac7de7-aa2e-4afb-a843-9b87f2eee58a_isrForCrosstalkSources_1031719_16.stderr parsl.dataflow.dflow INFO: Task 1838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78e268025a33c15ce1e365d27f164a29 with result from task 1838 parsl.dataflow.dflow DEBUG: Task 1838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/9f66dc9c-3aca-49a6-938d-74ad128dea42_isrForCrosstalkSources_1031699_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/9f66dc9c-3aca-49a6-938d-74ad128dea42_isrForCrosstalkSources_1031699_16.stderr parsl.dataflow.dflow INFO: Task 2182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68940d46fddd4caa950d02f700980d54 with result from task 2182 parsl.dataflow.dflow DEBUG: Task 2182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/749eebc9-e5e3-4a0a-9468-4eb6a1204c02_isrForCrosstalkSources_1031722_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/749eebc9-e5e3-4a0a-9468-4eb6a1204c02_isrForCrosstalkSources_1031722_19.stderr parsl.dataflow.dflow INFO: Task 1848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d42a6d1bdf6fba61dd5ffe6f63df4c28 with result from task 1848 parsl.dataflow.dflow DEBUG: Task 1848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/29ebfbc7-4b8d-4df8-a63a-e3ef67bc87f3_isrForCrosstalkSources_1031712_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/29ebfbc7-4b8d-4df8-a63a-e3ef67bc87f3_isrForCrosstalkSources_1031712_10.stderr parsl.dataflow.dflow INFO: Task 1693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b969fc8791169f8b58c6affb029870f9 with result from task 1693 parsl.dataflow.dflow DEBUG: Task 1693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/750c86ff-e63c-47cb-a6ba-1bb9d98a1736_isrForCrosstalkSources_1031738_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/750c86ff-e63c-47cb-a6ba-1bb9d98a1736_isrForCrosstalkSources_1031738_13.stderr parsl.dataflow.dflow INFO: Task 1723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd8674ed3c29cd2ceac57c4da54da6f0 with result from task 1723 parsl.dataflow.dflow DEBUG: Task 1723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/af2c9736-ccb7-4450-9de1-284d20bef359_isrForCrosstalkSources_1031727_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/af2c9736-ccb7-4450-9de1-284d20bef359_isrForCrosstalkSources_1031727_1.stderr parsl.dataflow.dflow INFO: Task 1903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76adfc94e0a9e3e30bab4e9c361f5276 with result from task 1903 parsl.dataflow.dflow DEBUG: Task 1903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/57bc79ea-b821-401f-a2e6-1a5a14599776_isrForCrosstalkSources_1031722_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/57bc79ea-b821-401f-a2e6-1a5a14599776_isrForCrosstalkSources_1031722_16.stderr parsl.dataflow.dflow INFO: Task 1399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1c22c70f95aa04708269f2f47c4c2ee with result from task 1399 parsl.dataflow.dflow DEBUG: Task 1399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/e8db40a5-3a92-424d-85e9-a7fd45a8bf39_isrForCrosstalkSources_1031735_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/e8db40a5-3a92-424d-85e9-a7fd45a8bf39_isrForCrosstalkSources_1031735_17.stderr parsl.dataflow.dflow INFO: Task 1725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ee57144c484ad4371f976257a0091ac with result from task 1725 parsl.dataflow.dflow DEBUG: Task 1725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e4890cf5-d366-4a82-84e0-954a927f81c0_isrForCrosstalkSources_1031707_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/e4890cf5-d366-4a82-84e0-954a927f81c0_isrForCrosstalkSources_1031707_16.stderr parsl.dataflow.dflow INFO: Task 1695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56aef0d6ff5ff7a1b773c61e0572d66b with result from task 1695 parsl.dataflow.dflow DEBUG: Task 1695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d2500e12-7889-4475-bd08-11c1611db1dc_isrForCrosstalkSources_1031738_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d2500e12-7889-4475-bd08-11c1611db1dc_isrForCrosstalkSources_1031738_61.stderr parsl.dataflow.dflow INFO: Task 1205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b29990cce2afc600cb0f6da83596e760 with result from task 1205 parsl.dataflow.dflow DEBUG: Task 1205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/75974c60-c5aa-457e-88fd-f9b6ba696336_isrForCrosstalkSources_1031723_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/75974c60-c5aa-457e-88fd-f9b6ba696336_isrForCrosstalkSources_1031723_55.stderr parsl.dataflow.dflow INFO: Task 1618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b76a7a65ce9fc10aa197e463a754e46 with result from task 1618 parsl.dataflow.dflow DEBUG: Task 1618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/f76da089-51b8-4a3e-adc6-43c990b58269_isrForCrosstalkSources_1031715_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/f76da089-51b8-4a3e-adc6-43c990b58269_isrForCrosstalkSources_1031715_14.stderr parsl.dataflow.dflow INFO: Task 1373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8496234f6d20bd0331b8a801c15c94f9 with result from task 1373 parsl.dataflow.dflow DEBUG: Task 1373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/19f9017d-55b2-4188-94bb-31fbbd95c52f_isrForCrosstalkSources_1031733_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/19f9017d-55b2-4188-94bb-31fbbd95c52f_isrForCrosstalkSources_1031733_5.stderr 2024-03-17 04:06:51 proc_lsst.multi:170 [INFO] [multi] job_id 17148167 proc_lsst.multi INFO: [multi] job_id 17148167 2024-03-17 04:06:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 164 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 164 2024-03-17 04:06:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148167 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148167 parsl.executors.status_handling DEBUG: Launched block 219 on executor multi with job ID 17148167 parsl.executors.status_handling INFO: Allocated block ID 220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-220 2024-03-17 04:06:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 2024-03-17 04:06:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 164 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 164 2024-03-17 04:06:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09e9a08d9ee4171af2c275f093c43036 with result from task 1872 parsl.dataflow.dflow DEBUG: Task 1872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/e087b8e0-ddc5-4934-a8a9-4b60f1edee55_isrForCrosstalkSources_1031708_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/e087b8e0-ddc5-4934-a8a9-4b60f1edee55_isrForCrosstalkSources_1031708_44.stderr parsl.dataflow.dflow INFO: Task 1358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 811c35e7e48d0c45eee1f723b069abab with result from task 1358 parsl.dataflow.dflow DEBUG: Task 1358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1b6c3c6a-1379-4da5-a553-abda853bc41c_isrForCrosstalkSources_1031711_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/1b6c3c6a-1379-4da5-a553-abda853bc41c_isrForCrosstalkSources_1031711_50.stderr parsl.dataflow.dflow INFO: Task 1840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77c78a3440ccd289c1aa9cc30e653998 with result from task 1840 parsl.dataflow.dflow DEBUG: Task 1840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e48e452b-3512-47c1-9d2f-1d9a8632995c_isrForCrosstalkSources_1031733_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/e48e452b-3512-47c1-9d2f-1d9a8632995c_isrForCrosstalkSources_1031733_27.stderr parsl.dataflow.dflow INFO: Task 1755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e863365c854a3ea8280b90e027056159 with result from task 1755 parsl.dataflow.dflow DEBUG: Task 1755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d04143f1-0eef-4cde-97df-98fb63d6ef52_isrForCrosstalkSources_1031726_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/d04143f1-0eef-4cde-97df-98fb63d6ef52_isrForCrosstalkSources_1031726_49.stderr parsl.dataflow.dflow INFO: Task 2188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44fccafda5f71bc7c84393d4c6ad7f05 with result from task 2188 parsl.dataflow.dflow DEBUG: Task 2188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a88eedd3-418d-4b3d-bfdc-8b502d412261_isrForCrosstalkSources_1031702_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a88eedd3-418d-4b3d-bfdc-8b502d412261_isrForCrosstalkSources_1031702_45.stderr parsl.dataflow.dflow INFO: Task 1750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbc81357af062930d56dc8e5e7efd939 with result from task 1750 parsl.dataflow.dflow DEBUG: Task 1750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/d80f102f-526d-4f85-994d-6de1c0433a0f_isrForCrosstalkSources_1031732_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/d80f102f-526d-4f85-994d-6de1c0433a0f_isrForCrosstalkSources_1031732_58.stderr parsl.dataflow.dflow INFO: Task 1756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b79a80f136a467649da04af19c0a85f5 with result from task 1756 parsl.dataflow.dflow DEBUG: Task 1756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/1c0c48f2-554c-4984-a287-1dab7f2aaa4e_isrForCrosstalkSources_1031734_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/1c0c48f2-554c-4984-a287-1dab7f2aaa4e_isrForCrosstalkSources_1031734_23.stderr parsl.dataflow.dflow INFO: Task 1677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5032f461ac67ddc8b4c8e80fc5c1b9d4 with result from task 1677 parsl.dataflow.dflow DEBUG: Task 1677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/acf738d2-7bea-4979-84ee-639e16eda996_isrForCrosstalkSources_1031716_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/acf738d2-7bea-4979-84ee-639e16eda996_isrForCrosstalkSources_1031716_51.stderr parsl.dataflow.dflow INFO: Task 1718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0726d6563695217bdc011265ae673dbb with result from task 1718 parsl.dataflow.dflow DEBUG: Task 1718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/6910db0b-a839-4042-9ea8-7b6e7fed13c6_isrForCrosstalkSources_1031703_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/6910db0b-a839-4042-9ea8-7b6e7fed13c6_isrForCrosstalkSources_1031703_35.stderr parsl.dataflow.dflow INFO: Task 2247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dde3a7f569fc7c5083b4406cebe7c61a with result from task 2247 parsl.dataflow.dflow DEBUG: Task 2247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b37e545a-1e5c-4e91-a8c9-f0e87071c881_isrForCrosstalkSources_1031728_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/b37e545a-1e5c-4e91-a8c9-f0e87071c881_isrForCrosstalkSources_1031728_41.stderr parsl.dataflow.dflow INFO: Task 1654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28db65cc070aa484c42304daf35e5638 with result from task 1654 parsl.dataflow.dflow DEBUG: Task 1654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3476349c-d8a0-454a-8e71-a0df66680ad2_isrForCrosstalkSources_1031738_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/3476349c-d8a0-454a-8e71-a0df66680ad2_isrForCrosstalkSources_1031738_44.stderr parsl.dataflow.dflow INFO: Task 1691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 600830826af2c55c35f9d3c676bf88ff with result from task 1691 parsl.dataflow.dflow DEBUG: Task 1691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/208063c6-106e-48ae-b312-914cfd099027_isrForCrosstalkSources_1031732_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/208063c6-106e-48ae-b312-914cfd099027_isrForCrosstalkSources_1031732_49.stderr 2024-03-17 04:06:52 proc_lsst.multi:170 [INFO] [multi] job_id 17148168 proc_lsst.multi INFO: [multi] job_id 17148168 2024-03-17 04:06:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 165 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 165 2024-03-17 04:06:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148168 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148168 parsl.executors.status_handling DEBUG: Launched block 220 on executor multi with job ID 17148168 parsl.executors.status_handling INFO: Allocated block ID 221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-221 2024-03-17 04:06:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 2024-03-17 04:06:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Task 1829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a0d81d2194c95677b1157f202010a96 with result from task 1829 proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 165 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 165 2024-03-17 04:06:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 1829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/cacf479f-a9b9-4fcd-9b84-50cddd9b38df_isrForCrosstalkSources_1031724_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/cacf479f-a9b9-4fcd-9b84-50cddd9b38df_isrForCrosstalkSources_1031724_42.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfbca1b33c43314be955777f525d85db with result from task 1352 parsl.dataflow.dflow DEBUG: Task 1352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/2ccfcd09-f06e-4a60-a96f-feaf1b331e1c_isrForCrosstalkSources_1031716_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/2ccfcd09-f06e-4a60-a96f-feaf1b331e1c_isrForCrosstalkSources_1031716_6.stderr parsl.dataflow.dflow INFO: Task 1673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ffed08838cdf5965956bb7c6be1bf9e with result from task 1673 parsl.dataflow.dflow DEBUG: Task 1673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/0295b3fd-f711-45b5-8206-376d683d07d9_isrForCrosstalkSources_1031704_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/0295b3fd-f711-45b5-8206-376d683d07d9_isrForCrosstalkSources_1031704_61.stderr parsl.dataflow.dflow INFO: Task 2005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa8143f4e0c795138495199f3d6d4835 with result from task 2005 parsl.dataflow.dflow DEBUG: Task 2005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/22d0eb95-6f15-442b-b43f-bddd5616f8ae_isrForCrosstalkSources_1031700_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/22d0eb95-6f15-442b-b43f-bddd5616f8ae_isrForCrosstalkSources_1031700_19.stderr parsl.dataflow.dflow INFO: Task 1742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0673c44fb39227588e0915f267bbe555 with result from task 1742 parsl.dataflow.dflow DEBUG: Task 1742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/79eb2091-1697-4a50-bc17-e480714583a7_isrForCrosstalkSources_1031704_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/79eb2091-1697-4a50-bc17-e480714583a7_isrForCrosstalkSources_1031704_48.stderr parsl.dataflow.dflow INFO: Task 1679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dd2f0ca70a9ac471d9da9da12eb50cd with result from task 1679 parsl.dataflow.dflow DEBUG: Task 1679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/aa2d9a40-565b-4b41-b727-dee692877b5b_isrForCrosstalkSources_1031719_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/aa2d9a40-565b-4b41-b727-dee692877b5b_isrForCrosstalkSources_1031719_53.stderr parsl.dataflow.dflow INFO: Task 1713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2baf6bd733af3d3b59d2a631f0aeadc3 with result from task 1713 parsl.dataflow.dflow DEBUG: Task 1713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9ac3e498-801d-4cf1-b386-e35abaa3ce33_isrForCrosstalkSources_1031729_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/9ac3e498-801d-4cf1-b386-e35abaa3ce33_isrForCrosstalkSources_1031729_6.stderr parsl.dataflow.dflow INFO: Task 2045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c7e7041afc2a66af4b494b22eb576c9 with result from task 2045 parsl.dataflow.dflow DEBUG: Task 2045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/dd416c02-eba8-4c05-bb83-b755e46afa36_isrForCrosstalkSources_1031728_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/dd416c02-eba8-4c05-bb83-b755e46afa36_isrForCrosstalkSources_1031728_40.stderr parsl.dataflow.dflow INFO: Task 1239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32104a1d9be7974523391fe0620830a4 with result from task 1239 parsl.dataflow.dflow DEBUG: Task 1239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/91ad8e44-58c5-47cc-b2fb-f86aa3343136_isrForCrosstalkSources_1031734_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/91ad8e44-58c5-47cc-b2fb-f86aa3343136_isrForCrosstalkSources_1031734_55.stderr parsl.dataflow.dflow INFO: Task 1712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a13281daae74d4743a5496e6581e1f6 with result from task 1712 parsl.dataflow.dflow DEBUG: Task 1712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2021aed1-1aa8-46ea-83cd-3bc8f9695e8b_isrForCrosstalkSources_1031703_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2021aed1-1aa8-46ea-83cd-3bc8f9695e8b_isrForCrosstalkSources_1031703_41.stderr parsl.dataflow.dflow INFO: Task 1683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0cb2ffbaf53ba5cd1492386052a8ff2 with result from task 1683 parsl.dataflow.dflow DEBUG: Task 1683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/786f6546-1239-406d-a1e2-95970423520e_isrForCrosstalkSources_1031722_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/786f6546-1239-406d-a1e2-95970423520e_isrForCrosstalkSources_1031722_59.stderr parsl.dataflow.dflow INFO: Task 2208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef1ac6cc2d7fda5d355cd9a2d3344289 with result from task 2208 parsl.dataflow.dflow DEBUG: Task 2208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a5e97407-67d2-4fb8-a932-ed042feb1eb5_isrForCrosstalkSources_1031701_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a5e97407-67d2-4fb8-a932-ed042feb1eb5_isrForCrosstalkSources_1031701_23.stderr parsl.dataflow.dflow INFO: Task 1716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b29f5244cd9703664a24c7498065cdf with result from task 1716 parsl.dataflow.dflow DEBUG: Task 1716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/792170c2-dd73-446f-bc4d-03bd68ed3256_isrForCrosstalkSources_1031713_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/792170c2-dd73-446f-bc4d-03bd68ed3256_isrForCrosstalkSources_1031713_5.stderr 2024-03-17 04:06:53 proc_lsst.multi:170 [INFO] [multi] job_id 17148169 proc_lsst.multi INFO: [multi] job_id 17148169 2024-03-17 04:06:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 166 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 166 2024-03-17 04:06:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148169 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148169 parsl.executors.status_handling DEBUG: Launched block 221 on executor multi with job ID 17148169 parsl.executors.status_handling INFO: Allocated block ID 222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-222 2024-03-17 04:06:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 2024-03-17 04:06:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 166 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 166 2024-03-17 04:06:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f8d088189523ba01cc0d14b29ecfa2d with result from task 1743 parsl.dataflow.dflow DEBUG: Task 1743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/93290f0f-7645-4b2c-ae8c-a0d3955a3ccf_isrForCrosstalkSources_1031699_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/93290f0f-7645-4b2c-ae8c-a0d3955a3ccf_isrForCrosstalkSources_1031699_30.stderr parsl.dataflow.dflow INFO: Task 1629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c72269d4ad5c199ba59bec036731a559 with result from task 1629 parsl.dataflow.dflow DEBUG: Task 1629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/43369357-1171-4f1d-986f-de8a14835bbc_isrForCrosstalkSources_1031718_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/43369357-1171-4f1d-986f-de8a14835bbc_isrForCrosstalkSources_1031718_10.stderr parsl.dataflow.dflow INFO: Task 1710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b89645e24ace82aeecd8f1bb7a175ab4 with result from task 1710 parsl.dataflow.dflow DEBUG: Task 1710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/83d75930-e5f7-4897-b675-a32fcf1bb107_isrForCrosstalkSources_1031706_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/83d75930-e5f7-4897-b675-a32fcf1bb107_isrForCrosstalkSources_1031706_29.stderr parsl.dataflow.dflow INFO: Task 1729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba4241c3d5347dfd10980fc1536e7ce0 with result from task 1729 parsl.dataflow.dflow DEBUG: Task 1729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/672c5c41-14d5-4308-80dd-5d67093dbdbe_isrForCrosstalkSources_1031718_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/672c5c41-14d5-4308-80dd-5d67093dbdbe_isrForCrosstalkSources_1031718_54.stderr parsl.dataflow.dflow INFO: Task 1692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 805eb0eaa689adebd6eb444a2747a398 with result from task 1692 parsl.dataflow.dflow DEBUG: Task 1692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/beb39741-be98-41ab-a22a-292b6e43081e_isrForCrosstalkSources_1031700_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/beb39741-be98-41ab-a22a-292b6e43081e_isrForCrosstalkSources_1031700_44.stderr parsl.dataflow.dflow INFO: Task 1696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2664083b53722a087720af4d40a7692b with result from task 1696 parsl.dataflow.dflow DEBUG: Task 1696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eeeeea7b-8fff-4645-947a-fb1065282358_isrForCrosstalkSources_1031719_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/eeeeea7b-8fff-4645-947a-fb1065282358_isrForCrosstalkSources_1031719_51.stderr parsl.dataflow.dflow INFO: Task 1343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ee9b40e7c68ca9a6bf136823c28b9f5 with result from task 1343 parsl.dataflow.dflow DEBUG: Task 1343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fb4fc5f6-fa96-4ce5-a485-a594599049ac_isrForCrosstalkSources_1031707_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/fb4fc5f6-fa96-4ce5-a485-a594599049ac_isrForCrosstalkSources_1031707_54.stderr parsl.dataflow.dflow INFO: Task 1863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa282c5918d4ae0aeac3796bc5d82b9d with result from task 1863 parsl.dataflow.dflow DEBUG: Task 1863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bf35a201-ea7c-4343-a803-f831dfdabdba_isrForCrosstalkSources_1031723_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bf35a201-ea7c-4343-a803-f831dfdabdba_isrForCrosstalkSources_1031723_44.stderr parsl.dataflow.dflow INFO: Task 1782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60ebae850f5bc060175eeda35c80cc0f with result from task 1782 parsl.dataflow.dflow DEBUG: Task 1782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e306d0c5-355a-4d0e-8857-51cfec600e2d_isrForCrosstalkSources_1031730_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/e306d0c5-355a-4d0e-8857-51cfec600e2d_isrForCrosstalkSources_1031730_39.stderr parsl.dataflow.dflow INFO: Task 1353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0b9d4e51b50b4a69d2b475e6549f5b8 with result from task 1353 parsl.dataflow.dflow DEBUG: Task 1353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/50127c9c-563c-40e9-932a-c73f444469ce_isrForCrosstalkSources_1031712_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/50127c9c-563c-40e9-932a-c73f444469ce_isrForCrosstalkSources_1031712_47.stderr parsl.dataflow.dflow INFO: Task 1751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b55acc2ead5d19663fcfddf39684250a with result from task 1751 parsl.dataflow.dflow DEBUG: Task 1751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fddf19bc-302f-40e3-b325-a5b8cc1bef16_isrForCrosstalkSources_1031732_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/fddf19bc-302f-40e3-b325-a5b8cc1bef16_isrForCrosstalkSources_1031732_30.stderr parsl.dataflow.dflow INFO: Task 1735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fe7620a0868e5d06ad4e13c71f6fe09 with result from task 1735 parsl.dataflow.dflow DEBUG: Task 1735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/69824f0f-e5b2-47ce-8504-532f571d77ac_isrForCrosstalkSources_1031722_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/69824f0f-e5b2-47ce-8504-532f571d77ac_isrForCrosstalkSources_1031722_1.stderr parsl.dataflow.dflow INFO: Task 1345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 113cf02f9465897f589bd70d574207cd with result from task 1345 parsl.dataflow.dflow DEBUG: Task 1345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7f95ab84-35ea-4ca1-af3c-4cfc54ff2bec_isrForCrosstalkSources_1031735_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/7f95ab84-35ea-4ca1-af3c-4cfc54ff2bec_isrForCrosstalkSources_1031735_48.stderr parsl.dataflow.dflow INFO: Task 1670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fde4dfe66255169dd7cf47b6361eb5f3 with result from task 1670 parsl.dataflow.dflow DEBUG: Task 1670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/ece996fb-9292-4eb2-8489-cd368b623fad_isrForCrosstalkSources_1031734_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/ece996fb-9292-4eb2-8489-cd368b623fad_isrForCrosstalkSources_1031734_31.stderr parsl.dataflow.dflow INFO: Task 2169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e84934c435346947281c3e7a58da3d7a with result from task 2169 parsl.dataflow.dflow DEBUG: Task 2169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/326807c7-bd57-4052-8612-02abce1859a7_isrForCrosstalkSources_1031701_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/326807c7-bd57-4052-8612-02abce1859a7_isrForCrosstalkSources_1031701_43.stderr 2024-03-17 04:06:53 proc_lsst.multi:170 [INFO] [multi] job_id 17148170 proc_lsst.multi INFO: [multi] job_id 17148170 2024-03-17 04:06:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 167 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 167 2024-03-17 04:06:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148170 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148170 parsl.executors.status_handling DEBUG: Launched block 222 on executor multi with job ID 17148170 parsl.executors.status_handling INFO: Allocated block ID 223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-223 2024-03-17 04:06:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 2024-03-17 04:06:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 167 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 167 2024-03-17 04:06:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1818 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization DEBUG: Storing app cache entry 455f571543e098423eac264684b87c09 with result from task 1818 parsl.dataflow.dflow DEBUG: Task 1818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/1c670202-391e-459f-aac5-50397ec04a5c_isrForCrosstalkSources_1031719_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/1c670202-391e-459f-aac5-50397ec04a5c_isrForCrosstalkSources_1031719_60.stderr parsl.dataflow.dflow INFO: Task 1868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7d4e7033ff26b1b80f551447507f7dc with result from task 1868 parsl.dataflow.dflow DEBUG: Task 1868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4e3f2c43-000a-4ecc-9d40-49dc67f121a1_isrForCrosstalkSources_1031717_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4e3f2c43-000a-4ecc-9d40-49dc67f121a1_isrForCrosstalkSources_1031717_50.stderr parsl.dataflow.dflow INFO: Task 1357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55a6ee47eb195e0cd67a247b3327d0e6 with result from task 1357 parsl.dataflow.dflow DEBUG: Task 1357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/963ae289-8e80-440e-a331-c349e79530b4_isrForCrosstalkSources_1031701_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/963ae289-8e80-440e-a331-c349e79530b4_isrForCrosstalkSources_1031701_62.stderr parsl.dataflow.dflow INFO: Task 1937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c909734a5e2cc487695ce0c587f2de6 with result from task 1937 parsl.dataflow.dflow DEBUG: Task 1937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/236fd25c-597d-40d0-a69f-826123177286_isrForCrosstalkSources_1031730_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/236fd25c-597d-40d0-a69f-826123177286_isrForCrosstalkSources_1031730_35.stderr parsl.dataflow.dflow INFO: Task 1790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ed643d53bc57bf8a6489b2015f6ba9d with result from task 1790 parsl.dataflow.dflow DEBUG: Task 1790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/565f8398-a2cf-4f68-b6bd-c054dc4c3b9d_isrForCrosstalkSources_1031720_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/565f8398-a2cf-4f68-b6bd-c054dc4c3b9d_isrForCrosstalkSources_1031720_56.stderr parsl.dataflow.dflow INFO: Task 1780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c5ca242957f7c9d0495bac83f71623f with result from task 1780 parsl.dataflow.dflow DEBUG: Task 1780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c7996ae4-3d99-4277-9eb4-9125a0c602df_isrForCrosstalkSources_1031715_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/c7996ae4-3d99-4277-9eb4-9125a0c602df_isrForCrosstalkSources_1031715_8.stderr parsl.dataflow.dflow INFO: Task 1953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cedbfe1454da755e2dc80106e28f9a6 with result from task 1953 parsl.dataflow.dflow DEBUG: Task 1953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c5b789ce-f159-4fee-a1d1-227b5f8879a1_isrForCrosstalkSources_1031708_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c5b789ce-f159-4fee-a1d1-227b5f8879a1_isrForCrosstalkSources_1031708_14.stderr parsl.dataflow.dflow INFO: Task 1949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5438856df981f799121175dda7bfb9e with result from task 1949 parsl.dataflow.dflow DEBUG: Task 1949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5dbd1431-ad2a-4357-925a-7cfe600a3e69_isrForCrosstalkSources_1031720_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/5dbd1431-ad2a-4357-925a-7cfe600a3e69_isrForCrosstalkSources_1031720_23.stderr parsl.dataflow.dflow INFO: Task 1748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31f4d084fc290e85ee175e28e9150c75 with result from task 1748 parsl.dataflow.dflow DEBUG: Task 1748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/61970c15-0c9a-48de-9ea8-addeb91f0f56_isrForCrosstalkSources_1031728_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/61970c15-0c9a-48de-9ea8-addeb91f0f56_isrForCrosstalkSources_1031728_16.stderr parsl.dataflow.dflow INFO: Task 1830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c958f279700a69e560b2b435a368d32 with result from task 1830 parsl.dataflow.dflow DEBUG: Task 1830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f9c3a6df-1694-48be-91da-a9aeb1693153_isrForCrosstalkSources_1031730_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/f9c3a6df-1694-48be-91da-a9aeb1693153_isrForCrosstalkSources_1031730_5.stderr parsl.dataflow.dflow INFO: Task 2006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a468516fafbdca16d88862d60d551a3 with result from task 2006 parsl.dataflow.dflow DEBUG: Task 2006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/929b5a66-0f93-4270-a1fb-9bfcb0b6466b_isrForCrosstalkSources_1031710_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/929b5a66-0f93-4270-a1fb-9bfcb0b6466b_isrForCrosstalkSources_1031710_12.stderr parsl.dataflow.dflow INFO: Task 1603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee69c2dd87794949c1a4a72e4cdab789 with result from task 1603 parsl.dataflow.dflow DEBUG: Task 1603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/218dffb0-a6a1-4263-964e-b7a579d2613e_isrForCrosstalkSources_1031724_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/218dffb0-a6a1-4263-964e-b7a579d2613e_isrForCrosstalkSources_1031724_60.stderr parsl.dataflow.dflow INFO: Task 1998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e4706d5eb4311c5b2d511e0668328b4 with result from task 1998 parsl.dataflow.dflow DEBUG: Task 1998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8cc1fb31-42e9-4f1a-85f3-86657d55010c_isrForCrosstalkSources_1031699_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8cc1fb31-42e9-4f1a-85f3-86657d55010c_isrForCrosstalkSources_1031699_21.stderr parsl.dataflow.dflow INFO: Task 1340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61c42b666746d75570a8d598ccec966a with result from task 1340 parsl.dataflow.dflow DEBUG: Task 1340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/6afcd16a-29dc-46d4-ba2d-faa9e4ab5a52_isrForCrosstalkSources_1031729_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/6afcd16a-29dc-46d4-ba2d-faa9e4ab5a52_isrForCrosstalkSources_1031729_58.stderr 2024-03-17 04:06:54 proc_lsst.multi:170 [INFO] [multi] job_id 17148171 proc_lsst.multi INFO: [multi] job_id 17148171 2024-03-17 04:06:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 168 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 168 2024-03-17 04:06:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148171 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148171 parsl.executors.status_handling DEBUG: Launched block 223 on executor multi with job ID 17148171 parsl.executors.status_handling INFO: Allocated block ID 224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-224 2024-03-17 04:06:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 2024-03-17 04:06:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 168 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 168 2024-03-17 04:06:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3d7caf0354f42cd7bc73b6f8fd8274d with result from task 1655 parsl.dataflow.dflow DEBUG: Task 1655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0846cd2c-7bb9-4040-8ea0-bbfe238a1337_isrForCrosstalkSources_1031732_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/0846cd2c-7bb9-4040-8ea0-bbfe238a1337_isrForCrosstalkSources_1031732_37.stderr parsl.dataflow.dflow INFO: Task 1731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f2e5478ad1b1ea0721e633c9a945e26 with result from task 1731 parsl.dataflow.dflow DEBUG: Task 1731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/933b37e0-50dc-4d44-9737-4ba7de128bdf_isrForCrosstalkSources_1031737_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/933b37e0-50dc-4d44-9737-4ba7de128bdf_isrForCrosstalkSources_1031737_59.stderr parsl.dataflow.dflow INFO: Task 1732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3e493815f7e9593d0c9b43d8ed0e37e with result from task 1732 parsl.dataflow.dflow DEBUG: Task 1732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/b3648bfd-e471-49f9-99ca-d078b684a638_isrForCrosstalkSources_1031730_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/b3648bfd-e471-49f9-99ca-d078b684a638_isrForCrosstalkSources_1031730_43.stderr parsl.dataflow.dflow INFO: Task 1951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91f8f4db5d4adaefed5bf4adba807e64 with result from task 1951 parsl.dataflow.dflow DEBUG: Task 1951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/21ce9300-2fae-4b8a-9ae4-63c195f8dce6_isrForCrosstalkSources_1031711_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/21ce9300-2fae-4b8a-9ae4-63c195f8dce6_isrForCrosstalkSources_1031711_12.stderr parsl.dataflow.dflow INFO: Task 1585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 740cc3b4f56656141e2d36632e9fca6d with result from task 1585 parsl.dataflow.dflow DEBUG: Task 1585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/167778b3-ec1c-4ad3-885b-499faa5fdd91_isrForCrosstalkSources_1031717_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/167778b3-ec1c-4ad3-885b-499faa5fdd91_isrForCrosstalkSources_1031717_52.stderr parsl.dataflow.dflow INFO: Task 1864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9213f298d335ec22171cb3da10e1f12 with result from task 1864 parsl.dataflow.dflow DEBUG: Task 1864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/44a8f164-fb83-4a4c-b93d-311b74d24597_isrForCrosstalkSources_1031710_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/44a8f164-fb83-4a4c-b93d-311b74d24597_isrForCrosstalkSources_1031710_16.stderr parsl.dataflow.dflow INFO: Task 1891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86790e45ce0d1f7750632f02bc503a8b with result from task 1891 parsl.dataflow.dflow DEBUG: Task 1891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/edf78914-0552-474c-9d2b-1c66fdf7ddd7_isrForCrosstalkSources_1031729_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/edf78914-0552-474c-9d2b-1c66fdf7ddd7_isrForCrosstalkSources_1031729_7.stderr parsl.dataflow.dflow INFO: Task 1904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2f27ef20ab0e1ec7b749c9889afc6a7 with result from task 1904 parsl.dataflow.dflow DEBUG: Task 1904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/ece6a0ca-1a07-4938-a229-c0824343ae89_isrForCrosstalkSources_1031704_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/ece6a0ca-1a07-4938-a229-c0824343ae89_isrForCrosstalkSources_1031704_6.stderr parsl.dataflow.dflow INFO: Task 2000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0e23bac909f807f5dc7b669915ebcde with result from task 2000 parsl.dataflow.dflow DEBUG: Task 2000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/d24b61e8-3f54-43aa-ac43-bad0dacbfc7d_isrForCrosstalkSources_1031736_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/d24b61e8-3f54-43aa-ac43-bad0dacbfc7d_isrForCrosstalkSources_1031736_51.stderr parsl.dataflow.dflow INFO: Task 1813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f587f4118bd28c2a9d798f65244aad04 with result from task 1813 parsl.dataflow.dflow DEBUG: Task 1813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/140a1f70-52d1-4f4e-9cb9-c544f4c19ce1_isrForCrosstalkSources_1031712_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/140a1f70-52d1-4f4e-9cb9-c544f4c19ce1_isrForCrosstalkSources_1031712_21.stderr parsl.dataflow.dflow INFO: Task 1614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4142742a14a9dc6bb2174d9791c19bf with result from task 1614 parsl.dataflow.dflow DEBUG: Task 1614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/721cd5d5-167f-43a7-b05d-7fd49c5c9739_isrForCrosstalkSources_1031702_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/721cd5d5-167f-43a7-b05d-7fd49c5c9739_isrForCrosstalkSources_1031702_5.stderr parsl.dataflow.dflow INFO: Task 1393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1278296b141dc4fcd677cdcd9d187f32 with result from task 1393 parsl.dataflow.dflow DEBUG: Task 1393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/7ec458af-0633-4006-89f0-02b0563b1178_isrForCrosstalkSources_1031722_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/7ec458af-0633-4006-89f0-02b0563b1178_isrForCrosstalkSources_1031722_55.stderr 2024-03-17 04:06:54 proc_lsst.multi:170 [INFO] [multi] job_id 17148172 proc_lsst.multi INFO: [multi] job_id 17148172 2024-03-17 04:06:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 169 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 169 2024-03-17 04:06:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148172 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148172 parsl.executors.status_handling DEBUG: Launched block 224 on executor multi with job ID 17148172 parsl.executors.status_handling INFO: Allocated block ID 225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-225 2024-03-17 04:06:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 2024-03-17 04:06:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 169 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 169 2024-03-17 04:06:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c218090225e552c76182dada79baa1d9 with result from task 1896 parsl.dataflow.dflow DEBUG: Task 1896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/e286aec0-c46f-4d96-858b-96e909a4ab95_isrForCrosstalkSources_1031708_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/e286aec0-c46f-4d96-858b-96e909a4ab95_isrForCrosstalkSources_1031708_4.stderr parsl.dataflow.dflow INFO: Task 1715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 013f63338c797ef50e0d6d999b734d93 with result from task 1715 parsl.dataflow.dflow DEBUG: Task 1715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7c97ff6c-49c7-4440-b559-f9d7079573c3_isrForCrosstalkSources_1031725_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7c97ff6c-49c7-4440-b559-f9d7079573c3_isrForCrosstalkSources_1031725_7.stderr parsl.dataflow.dflow INFO: Task 1837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db2e6b9e271a7d2e8103f9ce2f445822 with result from task 1837 parsl.dataflow.dflow DEBUG: Task 1837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/0e17f5d8-c0a7-471a-8830-b8fc2f851ec3_isrForCrosstalkSources_1031730_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/0e17f5d8-c0a7-471a-8830-b8fc2f851ec3_isrForCrosstalkSources_1031730_53.stderr parsl.dataflow.dflow INFO: Task 1844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e79643442d18f8e14dacf339181ddd7 with result from task 1844 parsl.dataflow.dflow DEBUG: Task 1844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f88a9278-3a8f-4609-a3d6-a21883de6726_isrForCrosstalkSources_1031735_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/f88a9278-3a8f-4609-a3d6-a21883de6726_isrForCrosstalkSources_1031735_5.stderr parsl.dataflow.dflow INFO: Task 1980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9bef014b5b96e22cad2375c39fdc00a with result from task 1980 parsl.dataflow.dflow DEBUG: Task 1980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/0bdee7f5-052a-4133-b15f-ad542aa2f02b_isrForCrosstalkSources_1031704_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/0bdee7f5-052a-4133-b15f-ad542aa2f02b_isrForCrosstalkSources_1031704_2.stderr parsl.dataflow.dflow INFO: Task 1728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b091b6f7566ebf0045e11649973203d with result from task 1728 parsl.dataflow.dflow DEBUG: Task 1728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d3c3604b-e0c6-4b49-aa58-05f951031b95_isrForCrosstalkSources_1031725_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d3c3604b-e0c6-4b49-aa58-05f951031b95_isrForCrosstalkSources_1031725_17.stderr parsl.dataflow.dflow INFO: Task 1866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9aac37aca0efb659ea25ead156e3d1e with result from task 1866 parsl.dataflow.dflow DEBUG: Task 1866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c0349430-ebe0-4d20-8878-bf038ff66ee2_isrForCrosstalkSources_1031712_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c0349430-ebe0-4d20-8878-bf038ff66ee2_isrForCrosstalkSources_1031712_18.stderr parsl.dataflow.dflow INFO: Task 1982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbc2f214b5e86f2b22b9323ecece822e with result from task 1982 parsl.dataflow.dflow DEBUG: Task 1982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8bd532d7-997a-47c7-bed5-0306c162db76_isrForCrosstalkSources_1031736_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/8bd532d7-997a-47c7-bed5-0306c162db76_isrForCrosstalkSources_1031736_7.stderr parsl.dataflow.dflow INFO: Task 2321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25d5413aa1d89283e16f894e36006bf2 with result from task 2321 parsl.dataflow.dflow DEBUG: Task 2321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/1b5a87fb-20c4-4baa-ab7c-3742a995a0cc_isrForCrosstalkSources_1031728_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/1b5a87fb-20c4-4baa-ab7c-3742a995a0cc_isrForCrosstalkSources_1031728_36.stderr parsl.dataflow.dflow INFO: Task 2127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 899ac16c355963586ad5bb8cf1a85ed7 with result from task 2127 parsl.dataflow.dflow DEBUG: Task 2127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/83a5f775-0cf5-4fb5-b210-5409be271de4_isrForCrosstalkSources_1031721_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/83a5f775-0cf5-4fb5-b210-5409be271de4_isrForCrosstalkSources_1031721_31.stderr 2024-03-17 04:06:55 proc_lsst.multi:170 [INFO] [multi] job_id 17148173 proc_lsst.multi INFO: [multi] job_id 17148173 2024-03-17 04:06:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 170 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 170 2024-03-17 04:06:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148173 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148173 parsl.executors.status_handling DEBUG: Launched block 225 on executor multi with job ID 17148173 parsl.executors.status_handling INFO: Allocated block ID 226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-226 2024-03-17 04:06:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 170 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 170 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dd5495541c8301fb7be6bfc6b244437 with result from task 1378 parsl.dataflow.dflow DEBUG: Task 1378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a21eafcb-195c-4ed9-ba06-1ac46b781a94_isrForCrosstalkSources_1031701_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/a21eafcb-195c-4ed9-ba06-1ac46b781a94_isrForCrosstalkSources_1031701_47.stderr parsl.dataflow.dflow INFO: Task 2008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec2ee5c695b8a0e5041bb4d087ebfe75 with result from task 2008 parsl.dataflow.dflow DEBUG: Task 2008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/ef4a3c28-a471-457f-84df-33c02ed94de0_isrForCrosstalkSources_1031703_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/ef4a3c28-a471-457f-84df-33c02ed94de0_isrForCrosstalkSources_1031703_44.stderr parsl.dataflow.dflow INFO: Task 1882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ec3eaf8f4cb98d1edee580bf39ec216 with result from task 1882 parsl.dataflow.dflow DEBUG: Task 1882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5a5ee25c-ef58-4371-bb7b-f7abf49a3ca2_isrForCrosstalkSources_1031714_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/5a5ee25c-ef58-4371-bb7b-f7abf49a3ca2_isrForCrosstalkSources_1031714_61.stderr 2024-03-17 04:06:55 proc_lsst.multi:170 [INFO] [multi] job_id 17148174 proc_lsst.multi INFO: [multi] job_id 17148174 2024-03-17 04:06:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 171 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 171 2024-03-17 04:06:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148174 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148174 parsl.executors.status_handling DEBUG: Launched block 226 on executor multi with job ID 17148174 parsl.executors.status_handling INFO: Allocated block ID 227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-227 2024-03-17 04:06:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 171 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 171 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dae868369fc510247c9aff68666d5d58 with result from task 1768 parsl.dataflow.dflow DEBUG: Task 1768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/b3b955a9-ae25-4f2a-b1a8-a84cd732fd28_isrForCrosstalkSources_1031736_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/b3b955a9-ae25-4f2a-b1a8-a84cd732fd28_isrForCrosstalkSources_1031736_42.stderr parsl.dataflow.dflow INFO: Task 1926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 899f1ff0722108b0bf5edc9376accc9f with result from task 1926 parsl.dataflow.dflow DEBUG: Task 1926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/7ebce880-feb9-46b9-9f46-fdfd8d8d8307_isrForCrosstalkSources_1031731_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/7ebce880-feb9-46b9-9f46-fdfd8d8d8307_isrForCrosstalkSources_1031731_53.stderr parsl.dataflow.dflow INFO: Task 1885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cc4fa8741e962e6a7ac9e4b2be2d71a with result from task 1885 parsl.dataflow.dflow DEBUG: Task 1885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/9a926a4e-7d77-4ffa-924d-e62c96cdf867_isrForCrosstalkSources_1031699_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/9a926a4e-7d77-4ffa-924d-e62c96cdf867_isrForCrosstalkSources_1031699_57.stderr parsl.dataflow.dflow INFO: Task 1841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5e0c7f8be3cf022eb61e446a785dbcb with result from task 1841 parsl.dataflow.dflow DEBUG: Task 1841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/c69c0137-9716-49b8-b22b-b9f9a2c4e9ae_isrForCrosstalkSources_1031736_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/c69c0137-9716-49b8-b22b-b9f9a2c4e9ae_isrForCrosstalkSources_1031736_57.stderr 2024-03-17 04:06:55 proc_lsst.multi:170 [INFO] [multi] job_id 17148175 proc_lsst.multi INFO: [multi] job_id 17148175 2024-03-17 04:06:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 172 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 172 2024-03-17 04:06:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148175 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148175 parsl.executors.status_handling DEBUG: Launched block 227 on executor multi with job ID 17148175 parsl.executors.status_handling INFO: Allocated block ID 228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-228 2024-03-17 04:06:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 172 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 172 2024-03-17 04:06:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e1b5c0da2b6b07f2807db38528b3843 with result from task 1820 parsl.dataflow.dflow DEBUG: Task 1820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/1deefce6-96b2-4fb5-864c-ec53dfb359af_isrForCrosstalkSources_1031710_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/1deefce6-96b2-4fb5-864c-ec53dfb359af_isrForCrosstalkSources_1031710_54.stderr parsl.dataflow.dflow INFO: Task 1638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9eaeecbee5512e80af8cb94992ee75e2 with result from task 1638 parsl.dataflow.dflow DEBUG: Task 1638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/250c38d1-173f-49e6-84f9-bafc8a314599_isrForCrosstalkSources_1031705_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/250c38d1-173f-49e6-84f9-bafc8a314599_isrForCrosstalkSources_1031705_60.stderr parsl.dataflow.dflow INFO: Task 1850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec30492254c8a639f33d0bf932e65326 with result from task 1850 parsl.dataflow.dflow DEBUG: Task 1850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/628d31f4-d66d-4f43-863a-0297c7315d97_isrForCrosstalkSources_1031722_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/628d31f4-d66d-4f43-863a-0297c7315d97_isrForCrosstalkSources_1031722_24.stderr parsl.dataflow.dflow INFO: Task 2020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 061a84fdfd0d8be63dd6281de9377725 with result from task 2020 parsl.dataflow.dflow DEBUG: Task 2020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/4fd0c0b5-9c80-468b-b28a-a3b03c3bc5c9_isrForCrosstalkSources_1031731_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/4fd0c0b5-9c80-468b-b28a-a3b03c3bc5c9_isrForCrosstalkSources_1031731_27.stderr 2024-03-17 04:06:56 proc_lsst.multi:170 [INFO] [multi] job_id 17148176 proc_lsst.multi INFO: [multi] job_id 17148176 2024-03-17 04:06:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 173 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 173 2024-03-17 04:06:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148176 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148176 parsl.executors.status_handling DEBUG: Launched block 228 on executor multi with job ID 17148176 parsl.executors.status_handling INFO: Allocated block ID 229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-229 2024-03-17 04:06:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 2024-03-17 04:06:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 173 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 173 2024-03-17 04:06:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22faf0a33935d046d2237fac33db5c43 with result from task 1372 parsl.dataflow.dflow DEBUG: Task 1372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f5dd3e42-5e07-4ef5-9135-289c3783998f_isrForCrosstalkSources_1031706_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f5dd3e42-5e07-4ef5-9135-289c3783998f_isrForCrosstalkSources_1031706_57.stderr parsl.dataflow.dflow INFO: Task 1397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d831b327813acfdcb176f2787b7c33db with result from task 1397 parsl.dataflow.dflow DEBUG: Task 1397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/77f6800a-06c0-42d2-af6e-c2c48ae79b28_isrForCrosstalkSources_1031713_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/77f6800a-06c0-42d2-af6e-c2c48ae79b28_isrForCrosstalkSources_1031713_55.stderr parsl.dataflow.dflow INFO: Task 1738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07908657d249cd2323726da5bf09f2f3 with result from task 1738 parsl.dataflow.dflow DEBUG: Task 1738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c714d5ba-e0ef-41b3-a5bc-d224b8f3e872_isrForCrosstalkSources_1031705_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/c714d5ba-e0ef-41b3-a5bc-d224b8f3e872_isrForCrosstalkSources_1031705_50.stderr parsl.dataflow.dflow INFO: Task 2204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29f3da789f2eb85c7ccb26ccacc0a955 with result from task 2204 parsl.dataflow.dflow DEBUG: Task 2204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e88be92b-e68c-43b3-87d5-8b399dc7dec1_isrForCrosstalkSources_1031728_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/e88be92b-e68c-43b3-87d5-8b399dc7dec1_isrForCrosstalkSources_1031728_33.stderr parsl.dataflow.dflow INFO: Task 1995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71c6197a2f9be4c0013e46e57e3280ec with result from task 1995 parsl.dataflow.dflow DEBUG: Task 1995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/e4b24a93-b364-4717-b7f2-21b468cf8107_isrForCrosstalkSources_1031721_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/e4b24a93-b364-4717-b7f2-21b468cf8107_isrForCrosstalkSources_1031721_29.stderr parsl.dataflow.dflow INFO: Task 1832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e05897ddae28504ad3469e8f3c711e74 with result from task 1832 parsl.dataflow.dflow DEBUG: Task 1832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/a27c4a77-9e47-4b00-9169-56267f623f5b_isrForCrosstalkSources_1031708_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/a27c4a77-9e47-4b00-9169-56267f623f5b_isrForCrosstalkSources_1031708_48.stderr parsl.dataflow.dflow INFO: Task 1771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ae5820cb1e876371eb981e2a9484398 with result from task 1771 parsl.dataflow.dflow DEBUG: Task 1771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/6c46c321-0d8f-4422-8ea9-0dcc13191584_isrForCrosstalkSources_1031722_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/6c46c321-0d8f-4422-8ea9-0dcc13191584_isrForCrosstalkSources_1031722_18.stderr parsl.dataflow.dflow INFO: Task 1676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06be6eabce1ac4547199f0533e8ea2df with result from task 1676 parsl.dataflow.dflow DEBUG: Task 1676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f58b8b8e-ae58-4f95-b4d4-4ef43211b89e_isrForCrosstalkSources_1031703_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/f58b8b8e-ae58-4f95-b4d4-4ef43211b89e_isrForCrosstalkSources_1031703_61.stderr 2024-03-17 04:06:56 proc_lsst.multi:170 [INFO] [multi] job_id 17148177 proc_lsst.multi INFO: [multi] job_id 17148177 2024-03-17 04:06:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 174 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 174 2024-03-17 04:06:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148177 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148177 parsl.executors.status_handling DEBUG: Launched block 229 on executor multi with job ID 17148177 parsl.executors.status_handling INFO: Allocated block ID 230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-230 2024-03-17 04:06:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 2024-03-17 04:06:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 174 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 174 2024-03-17 04:06:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c88c05dea31144edf556608036805e8a with result from task 1865 parsl.dataflow.dflow DEBUG: Task 1865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2c1305e5-83e9-4cd6-a6e1-1dca25675155_isrForCrosstalkSources_1031734_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2c1305e5-83e9-4cd6-a6e1-1dca25675155_isrForCrosstalkSources_1031734_12.stderr parsl.dataflow.dflow INFO: Task 1909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b354750ceaf52777d303a6f63f8c55ca with result from task 1909 parsl.dataflow.dflow DEBUG: Task 1909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/aa8eeb14-0fb7-42b8-af15-258aa8bea2ab_isrForCrosstalkSources_1031724_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/aa8eeb14-0fb7-42b8-af15-258aa8bea2ab_isrForCrosstalkSources_1031724_35.stderr parsl.dataflow.dflow INFO: Task 2026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 488e16366b13d9e3d559d3bb56bf7bea with result from task 2026 parsl.dataflow.dflow DEBUG: Task 2026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/87131a1a-56ec-4836-b1db-7baae0e9d87d_isrForCrosstalkSources_1031702_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/87131a1a-56ec-4836-b1db-7baae0e9d87d_isrForCrosstalkSources_1031702_17.stderr parsl.dataflow.dflow INFO: Task 1700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd1a590d65ccaeb1822586e6d1394b30 with result from task 1700 parsl.dataflow.dflow DEBUG: Task 1700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fbfe3858-46aa-41e9-9d1b-2efef636a63d_isrForCrosstalkSources_1031700_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fbfe3858-46aa-41e9-9d1b-2efef636a63d_isrForCrosstalkSources_1031700_31.stderr parsl.dataflow.dflow INFO: Task 1855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b34cd289f1b86aed48472b52a3a2ff2e with result from task 1855 parsl.dataflow.dflow DEBUG: Task 1855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1785112d-d1db-47ad-86a8-012a7b7a3a68_isrForCrosstalkSources_1031713_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/1785112d-d1db-47ad-86a8-012a7b7a3a68_isrForCrosstalkSources_1031713_26.stderr parsl.dataflow.dflow INFO: Task 2339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a10c0824d104eef31f9669e99d6b323 with result from task 2339 parsl.dataflow.dflow DEBUG: Task 2339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e5cb5e25-a91a-488d-9647-4feb345f39a7_isrForCrosstalkSources_1031703_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/e5cb5e25-a91a-488d-9647-4feb345f39a7_isrForCrosstalkSources_1031703_14.stderr parsl.dataflow.dflow INFO: Task 1988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 400e3875064e474c8320a1448f661090 with result from task 1988 parsl.dataflow.dflow DEBUG: Task 1988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/74634bb6-9dee-49f1-9c37-c01526490a44_isrForCrosstalkSources_1031710_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/74634bb6-9dee-49f1-9c37-c01526490a44_isrForCrosstalkSources_1031710_4.stderr 2024-03-17 04:06:57 proc_lsst.multi:170 [INFO] [multi] job_id 17148178 proc_lsst.multi INFO: [multi] job_id 17148178 2024-03-17 04:06:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 175 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 175 2024-03-17 04:06:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148178 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148178 parsl.executors.status_handling DEBUG: Launched block 230 on executor multi with job ID 17148178 parsl.executors.status_handling INFO: Allocated block ID 231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-231 2024-03-17 04:06:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 175 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 175 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb0d573b6a3804e7ac738e4bdaf3c133 with result from task 2310 parsl.dataflow.dflow DEBUG: Task 2310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9d2d0e35-bc6b-484b-a5b8-92a84d681b88_isrForCrosstalkSources_1031733_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/9d2d0e35-bc6b-484b-a5b8-92a84d681b88_isrForCrosstalkSources_1031733_47.stderr parsl.dataflow.dflow INFO: Task 2313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efe3df1f2e1b10e6accdd21efd0392b5 with result from task 2313 parsl.dataflow.dflow DEBUG: Task 2313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/609a9c14-39e5-4ae2-86b2-6e9a78b5a6e2_isrForCrosstalkSources_1031713_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/609a9c14-39e5-4ae2-86b2-6e9a78b5a6e2_isrForCrosstalkSources_1031713_44.stderr parsl.dataflow.dflow INFO: Task 2304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c941b669d2adbfb27cb7366c510e238 with result from task 2304 parsl.dataflow.dflow DEBUG: Task 2304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/89a68929-d11f-4b65-961d-fa28381982ae_isrForCrosstalkSources_1031737_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/89a68929-d11f-4b65-961d-fa28381982ae_isrForCrosstalkSources_1031737_14.stderr parsl.dataflow.dflow INFO: Task 1984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd6e8c6b7bdcb7e1dbfb1d520d241a91 with result from task 1984 parsl.dataflow.dflow DEBUG: Task 1984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/46b1d99f-26df-428e-a981-3a8289c9c554_isrForCrosstalkSources_1031716_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/46b1d99f-26df-428e-a981-3a8289c9c554_isrForCrosstalkSources_1031716_33.stderr parsl.dataflow.dflow INFO: Task 1897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3600eeb1b51dc18ac543e4b90085dea with result from task 1897 parsl.dataflow.dflow DEBUG: Task 1897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/5c951d22-66b4-461b-8623-7b3c86ed4790_isrForCrosstalkSources_1031722_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/5c951d22-66b4-461b-8623-7b3c86ed4790_isrForCrosstalkSources_1031722_62.stderr parsl.dataflow.dflow INFO: Task 1794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c72bc83daab4c6335194d27b52bb584b with result from task 1794 parsl.dataflow.dflow DEBUG: Task 1794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7f107a6f-dd7e-4a93-a745-baf209966257_isrForCrosstalkSources_1031725_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7f107a6f-dd7e-4a93-a745-baf209966257_isrForCrosstalkSources_1031725_59.stderr parsl.dataflow.dflow INFO: Task 1938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 554d6fd239be2b4f19eb839b033d027f with result from task 1938 parsl.dataflow.dflow DEBUG: Task 1938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/086812ca-ab0d-4aad-8881-bf666cf7c3a7_isrForCrosstalkSources_1031737_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/086812ca-ab0d-4aad-8881-bf666cf7c3a7_isrForCrosstalkSources_1031737_12.stderr 2024-03-17 04:06:57 proc_lsst.multi:170 [INFO] [multi] job_id 17148179 proc_lsst.multi INFO: [multi] job_id 17148179 2024-03-17 04:06:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 176 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 176 2024-03-17 04:06:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148179 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148179 parsl.executors.status_handling DEBUG: Launched block 231 on executor multi with job ID 17148179 parsl.executors.status_handling INFO: Allocated block ID 232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-232 2024-03-17 04:06:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 parsl.dataflow.dflow INFO: Task 1852 completed (launched -> exec_done) proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 parsl.dataflow.memoization DEBUG: Storing app cache entry 53eaf6102181195f4b19e009a3616fe7 with result from task 1852 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow DEBUG: Task 1852 checkpointed proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 176 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 176 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.dflow INFO: Standard output for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/359089e1-e7cc-4ed6-818b-632cb00c0113_isrForCrosstalkSources_1031709_15.stdout parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.dflow INFO: Standard error for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/359089e1-e7cc-4ed6-818b-632cb00c0113_isrForCrosstalkSources_1031709_15.stderr parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8aa39df45a99b72a57e22e3a13dfd307 with result from task 2348 parsl.dataflow.dflow DEBUG: Task 2348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ed4208fd-0b5c-4aa9-a6c5-62315321e8d8_isrForCrosstalkSources_1031699_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ed4208fd-0b5c-4aa9-a6c5-62315321e8d8_isrForCrosstalkSources_1031699_24.stderr parsl.dataflow.dflow INFO: Task 1856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17f3dc376cdfa8e7c835ac8ccc0032da with result from task 1856 parsl.dataflow.dflow DEBUG: Task 1856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/3eabd279-ea50-4846-872f-3dba0a09b59d_isrForCrosstalkSources_1031722_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/3eabd279-ea50-4846-872f-3dba0a09b59d_isrForCrosstalkSources_1031722_27.stderr parsl.dataflow.dflow INFO: Task 1853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f37decbaced2b38431600ce88ed3c82a with result from task 1853 parsl.dataflow.dflow DEBUG: Task 1853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7ced5873-6df6-48d8-a380-d1d9231ad0f1_isrForCrosstalkSources_1031710_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7ced5873-6df6-48d8-a380-d1d9231ad0f1_isrForCrosstalkSources_1031710_28.stderr parsl.dataflow.dflow INFO: Task 2217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a6730c919f46e2ef3e94117197012cd with result from task 2217 parsl.dataflow.dflow DEBUG: Task 2217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fb92c31a-eee5-4df7-b1d8-9f0ad3742cc9_isrForCrosstalkSources_1031700_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/fb92c31a-eee5-4df7-b1d8-9f0ad3742cc9_isrForCrosstalkSources_1031700_34.stderr parsl.dataflow.dflow INFO: Task 1950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7a318e3f2dc60b82b516b269413f9d0 with result from task 1950 parsl.dataflow.dflow DEBUG: Task 1950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/9dbb4d1b-3a3a-494a-8063-4908e00daec5_isrForCrosstalkSources_1031734_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/9dbb4d1b-3a3a-494a-8063-4908e00daec5_isrForCrosstalkSources_1031734_45.stderr parsl.dataflow.dflow INFO: Task 2173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ac4d6bf6da4ab5521810def17e266e0 with result from task 2173 parsl.dataflow.dflow DEBUG: Task 2173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/672fb773-80f4-4a8a-b22d-30a954a4c7e9_isrForCrosstalkSources_1031701_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/672fb773-80f4-4a8a-b22d-30a954a4c7e9_isrForCrosstalkSources_1031701_54.stderr parsl.dataflow.dflow INFO: Task 1992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 053b81029a2eb459cbd80730b5c1a7d6 with result from task 1992 parsl.dataflow.dflow DEBUG: Task 1992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d5c09e2a-0f6f-4e4a-90d6-59ccebb6076a_isrForCrosstalkSources_1031709_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d5c09e2a-0f6f-4e4a-90d6-59ccebb6076a_isrForCrosstalkSources_1031709_62.stderr parsl.dataflow.dflow INFO: Task 2190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae247c8debed6efdee51ffc2adc6260b with result from task 2190 parsl.dataflow.dflow DEBUG: Task 2190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d197296f-b38c-4c86-b445-9fe1001bb815_isrForCrosstalkSources_1031701_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/d197296f-b38c-4c86-b445-9fe1001bb815_isrForCrosstalkSources_1031701_14.stderr parsl.dataflow.dflow INFO: Task 2202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry faf4f75d58a6d54f448bee608f7b16a8 with result from task 2202 parsl.dataflow.dflow DEBUG: Task 2202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8525da9d-debb-4e9c-b789-73606c83a78a_isrForCrosstalkSources_1031719_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8525da9d-debb-4e9c-b789-73606c83a78a_isrForCrosstalkSources_1031719_13.stderr parsl.dataflow.dflow INFO: Task 2318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e326489dfd6fe73d2f89f4ae82f16fb with result from task 2318 parsl.dataflow.dflow DEBUG: Task 2318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c8b700b1-d645-4fea-b84b-7d9617745e96_isrForCrosstalkSources_1031710_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c8b700b1-d645-4fea-b84b-7d9617745e96_isrForCrosstalkSources_1031710_22.stderr parsl.dataflow.dflow INFO: Task 1919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb661fb9b7f0c9ceed5c3adf9a7b75b3 with result from task 1919 parsl.dataflow.dflow DEBUG: Task 1919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3bd2023b-3a21-4b82-ad1d-d8a2163cdcb0_isrForCrosstalkSources_1031713_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3bd2023b-3a21-4b82-ad1d-d8a2163cdcb0_isrForCrosstalkSources_1031713_31.stderr parsl.dataflow.dflow INFO: Task 2062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a3d55067d3905f98604667821eb2c4f with result from task 2062 parsl.dataflow.dflow DEBUG: Task 2062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/97b3a1de-1ae6-422b-88ee-a96b7d20693c_isrForCrosstalkSources_1031705_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/97b3a1de-1ae6-422b-88ee-a96b7d20693c_isrForCrosstalkSources_1031705_58.stderr parsl.dataflow.dflow INFO: Task 2233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b277a76e84cd1545c70c1378bbcbe734 with result from task 2233 parsl.dataflow.dflow DEBUG: Task 2233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d8060273-2360-493e-8612-25ca1d2b32e0_isrForCrosstalkSources_1031709_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/d8060273-2360-493e-8612-25ca1d2b32e0_isrForCrosstalkSources_1031709_25.stderr 2024-03-17 04:06:57 proc_lsst.multi:170 [INFO] [multi] job_id 17148180 proc_lsst.multi INFO: [multi] job_id 17148180 2024-03-17 04:06:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 177 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 177 2024-03-17 04:06:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148180 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148180 parsl.executors.status_handling DEBUG: Launched block 232 on executor multi with job ID 17148180 parsl.executors.status_handling INFO: Allocated block ID 233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-233 2024-03-17 04:06:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-233 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Task 1777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5b5f3d67e065589191061d941ceaeb7 with result from task 1777 proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 177 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 177 2024-03-17 04:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 1777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c3864046-2254-486c-b14b-4adae7823330_isrForCrosstalkSources_1031727_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c3864046-2254-486c-b14b-4adae7823330_isrForCrosstalkSources_1031727_4.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f3290d5592bc7e32e627993ef678470 with result from task 2133 parsl.dataflow.dflow DEBUG: Task 2133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/715a59eb-0a9e-406c-b843-25786c2c1844_isrForCrosstalkSources_1031701_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/715a59eb-0a9e-406c-b843-25786c2c1844_isrForCrosstalkSources_1031701_25.stderr parsl.dataflow.dflow INFO: Task 1791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c2bb3b4385240ee794d63000253ddd4 with result from task 1791 parsl.dataflow.dflow DEBUG: Task 1791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/76823704-0050-4b0a-a0ae-e17371633e87_isrForCrosstalkSources_1031729_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/76823704-0050-4b0a-a0ae-e17371633e87_isrForCrosstalkSources_1031729_54.stderr parsl.dataflow.dflow INFO: Task 2025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7bd9a12a05dfd6ec4b580cd47ba63ba with result from task 2025 parsl.dataflow.dflow DEBUG: Task 2025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/43fd63b3-22e8-452a-b6c7-c6efd6a730ec_isrForCrosstalkSources_1031709_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/43fd63b3-22e8-452a-b6c7-c6efd6a730ec_isrForCrosstalkSources_1031709_11.stderr parsl.dataflow.dflow INFO: Task 2342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 121101f70b93bdbf9f1b247f3a9412c1 with result from task 2342 parsl.dataflow.dflow DEBUG: Task 2342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f55062d8-60bb-437d-b2a2-76525e77a227_isrForCrosstalkSources_1031716_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f55062d8-60bb-437d-b2a2-76525e77a227_isrForCrosstalkSources_1031716_30.stderr parsl.dataflow.dflow INFO: Task 2023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f8a7a89280c375e02415218664fc2a9 with result from task 2023 parsl.dataflow.dflow DEBUG: Task 2023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/986953fe-7ed5-43fa-bc15-68ea4568ba4d_isrForCrosstalkSources_1031710_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/986953fe-7ed5-43fa-bc15-68ea4568ba4d_isrForCrosstalkSources_1031710_14.stderr parsl.dataflow.dflow INFO: Task 2307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a01e1679734d0fa944652e6aeb3009b with result from task 2307 parsl.dataflow.dflow DEBUG: Task 2307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/80e84c44-4210-4318-8f9e-96ab28d6a51e_isrForCrosstalkSources_1031720_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/80e84c44-4210-4318-8f9e-96ab28d6a51e_isrForCrosstalkSources_1031720_15.stderr parsl.dataflow.dflow INFO: Task 2138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe3d0cc922f353e05c527a3f21f175bd with result from task 2138 parsl.dataflow.dflow DEBUG: Task 2138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/5969260f-07c1-4e3e-b5ca-4c3779bb09e5_isrForCrosstalkSources_1031729_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/5969260f-07c1-4e3e-b5ca-4c3779bb09e5_isrForCrosstalkSources_1031729_21.stderr parsl.dataflow.dflow INFO: Task 1875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ef2277cf001f2ff36cad9096d5f2077 with result from task 1875 parsl.dataflow.dflow DEBUG: Task 1875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7e59fdf7-1511-49e0-8cad-7b2eaee3e915_isrForCrosstalkSources_1031710_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7e59fdf7-1511-49e0-8cad-7b2eaee3e915_isrForCrosstalkSources_1031710_42.stderr parsl.dataflow.dflow INFO: Task 2172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 618356e019f570a639d0b8280f78f1d8 with result from task 2172 parsl.dataflow.dflow DEBUG: Task 2172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/cb666c2e-874c-493b-977b-239691945246_isrForCrosstalkSources_1031720_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/cb666c2e-874c-493b-977b-239691945246_isrForCrosstalkSources_1031720_25.stderr parsl.dataflow.dflow INFO: Task 2004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe5be2af08701d5ca94c000a0fe5763 with result from task 2004 parsl.dataflow.dflow DEBUG: Task 2004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8b5d2438-d885-426d-8628-b17fb5cd2682_isrForCrosstalkSources_1031719_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/8b5d2438-d885-426d-8628-b17fb5cd2682_isrForCrosstalkSources_1031719_1.stderr parsl.dataflow.dflow INFO: Task 1911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5acfe2a7ef24f0e6d12b8eace31630e3 with result from task 1911 parsl.dataflow.dflow DEBUG: Task 1911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/119048a3-6545-48f1-a35b-75e05240910e_isrForCrosstalkSources_1031722_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/119048a3-6545-48f1-a35b-75e05240910e_isrForCrosstalkSources_1031722_51.stderr parsl.dataflow.dflow INFO: Task 1764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2d36b378a7c4cf33e5aedee2ed9408e with result from task 1764 parsl.dataflow.dflow DEBUG: Task 1764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b8e6f2f4-e085-41c0-a10d-aecad19fc867_isrForCrosstalkSources_1031714_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b8e6f2f4-e085-41c0-a10d-aecad19fc867_isrForCrosstalkSources_1031714_30.stderr parsl.dataflow.dflow INFO: Task 1779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dca2c2d88edf5fb3b1814a583f736a08 with result from task 1779 parsl.dataflow.dflow DEBUG: Task 1779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/dc992c28-4187-4d8b-bad9-87b8feb90a37_isrForCrosstalkSources_1031699_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/dc992c28-4187-4d8b-bad9-87b8feb90a37_isrForCrosstalkSources_1031699_31.stderr parsl.dataflow.dflow INFO: Task 2098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69edd51cab1a0cc1cb5cc6acffca5475 with result from task 2098 parsl.dataflow.dflow DEBUG: Task 2098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/eff85d48-05e5-498c-9e8c-754a4e52eea6_isrForCrosstalkSources_1031699_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/eff85d48-05e5-498c-9e8c-754a4e52eea6_isrForCrosstalkSources_1031699_34.stderr parsl.dataflow.dflow INFO: Task 2002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1344c2d527f06dfff95724c27d5931c with result from task 2002 parsl.dataflow.dflow DEBUG: Task 2002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f4f0c0c3-1803-4dfa-a458-66ca078169fb_isrForCrosstalkSources_1031716_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/f4f0c0c3-1803-4dfa-a458-66ca078169fb_isrForCrosstalkSources_1031716_56.stderr parsl.dataflow.dflow INFO: Task 1776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52d485ffa9960b4af55084606917e565 with result from task 1776 parsl.dataflow.dflow DEBUG: Task 1776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/55c59bbd-649d-470c-be23-11d1b483de4b_isrForCrosstalkSources_1031729_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/55c59bbd-649d-470c-be23-11d1b483de4b_isrForCrosstalkSources_1031729_9.stderr parsl.dataflow.dflow INFO: Task 1808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db9f3507b2a51039dc99a134de7e79e2 with result from task 1808 parsl.dataflow.dflow DEBUG: Task 1808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/20d336fa-37e1-4889-8a49-559ba89c32f6_isrForCrosstalkSources_1031732_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/20d336fa-37e1-4889-8a49-559ba89c32f6_isrForCrosstalkSources_1031732_43.stderr 2024-03-17 04:06:58 proc_lsst.multi:170 [INFO] [multi] job_id 17148181 proc_lsst.multi INFO: [multi] job_id 17148181 2024-03-17 04:06:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 178 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 178 2024-03-17 04:06:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148181 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148181 parsl.executors.status_handling DEBUG: Launched block 233 on executor multi with job ID 17148181 parsl.executors.status_handling INFO: Allocated block ID 234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-234 2024-03-17 04:06:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-234 2024-03-17 04:06:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:06:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:06:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:06:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Task 2316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed1858ea1fdec3235e5a05fa84820ba8 with result from task 2316 proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:06:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 178 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 178 2024-03-17 04:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:06:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 2316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/cb3231f2-6f79-4338-a680-d667d44fa422_isrForCrosstalkSources_1031729_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/cb3231f2-6f79-4338-a680-d667d44fa422_isrForCrosstalkSources_1031729_37.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 963d90e8c14f1900a286d229bf38a5b1 with result from task 2061 parsl.dataflow.dflow DEBUG: Task 2061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/0ca97749-eb33-43c4-8a78-b31633281c95_isrForCrosstalkSources_1031733_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/0ca97749-eb33-43c4-8a78-b31633281c95_isrForCrosstalkSources_1031733_8.stderr parsl.dataflow.dflow INFO: Task 2240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b5c05b640d1dadde2a02508a20c84ac with result from task 2240 parsl.dataflow.dflow DEBUG: Task 2240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/0c99e524-1a69-4cc1-a29b-419774331711_isrForCrosstalkSources_1031726_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/0c99e524-1a69-4cc1-a29b-419774331711_isrForCrosstalkSources_1031726_26.stderr parsl.dataflow.dflow INFO: Task 2291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 551ac8921b691b5d0d338b91b12456e4 with result from task 2291 parsl.dataflow.dflow DEBUG: Task 2291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/7c0e4fa1-52c0-4018-badf-b3f97585fcee_isrForCrosstalkSources_1031737_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/7c0e4fa1-52c0-4018-badf-b3f97585fcee_isrForCrosstalkSources_1031737_32.stderr parsl.dataflow.dflow INFO: Task 2358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 986a1dfd3ce6d3ebf85e6540fc1bc916 with result from task 2358 parsl.dataflow.dflow DEBUG: Task 2358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/3e8a2305-7e5c-493d-91b5-e0072e705492_isrForCrosstalkSources_1031730_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/3e8a2305-7e5c-493d-91b5-e0072e705492_isrForCrosstalkSources_1031730_4.stderr parsl.dataflow.dflow INFO: Task 1701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0c41314369084a267444070efe2b7ee with result from task 1701 parsl.dataflow.dflow DEBUG: Task 1701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cbc8d656-086c-4bda-b759-350e2099b6c3_isrForCrosstalkSources_1031728_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/cbc8d656-086c-4bda-b759-350e2099b6c3_isrForCrosstalkSources_1031728_62.stderr parsl.dataflow.dflow INFO: Task 2203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27f554581fb15a5b3f129cbd78ef0b7d with result from task 2203 parsl.dataflow.dflow DEBUG: Task 2203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/157743bb-34c5-4e12-8b41-666f0a846bfa_isrForCrosstalkSources_1031721_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/157743bb-34c5-4e12-8b41-666f0a846bfa_isrForCrosstalkSources_1031721_50.stderr parsl.dataflow.dflow INFO: Task 1804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e913b9250c9d617e7b5c15fbb6abd8b5 with result from task 1804 parsl.dataflow.dflow DEBUG: Task 1804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9427d0fc-0fa6-42f5-be12-735956fb7f73_isrForCrosstalkSources_1031730_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/9427d0fc-0fa6-42f5-be12-735956fb7f73_isrForCrosstalkSources_1031730_45.stderr parsl.dataflow.dflow INFO: Task 2060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85ffcf2d397c5de9c15c30574de54181 with result from task 2060 parsl.dataflow.dflow DEBUG: Task 2060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8736a8eb-3bcb-4164-8ac3-f5d3b8796f73_isrForCrosstalkSources_1031703_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8736a8eb-3bcb-4164-8ac3-f5d3b8796f73_isrForCrosstalkSources_1031703_8.stderr parsl.dataflow.dflow INFO: Task 2021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 956e86f57372d42f29bca3ee010adae6 with result from task 2021 parsl.dataflow.dflow DEBUG: Task 2021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/8b7a9a36-3c66-42f8-b10f-799e12724254_isrForCrosstalkSources_1031718_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/8b7a9a36-3c66-42f8-b10f-799e12724254_isrForCrosstalkSources_1031718_4.stderr parsl.dataflow.dflow INFO: Task 1737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91281e2a945b3b7fa13ac0e2d4222af4 with result from task 1737 parsl.dataflow.dflow DEBUG: Task 1737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/dc63a10e-8662-45b3-bb37-0650d55cc86b_isrForCrosstalkSources_1031728_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/dc63a10e-8662-45b3-bb37-0650d55cc86b_isrForCrosstalkSources_1031728_52.stderr parsl.dataflow.dflow INFO: Task 1719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab14850294e8c9b8395d11a376bd2604 with result from task 1719 parsl.dataflow.dflow DEBUG: Task 1719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/55bb91cc-c320-4018-888d-a838b051d51e_isrForCrosstalkSources_1031711_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/55bb91cc-c320-4018-888d-a838b051d51e_isrForCrosstalkSources_1031711_37.stderr parsl.dataflow.dflow INFO: Task 1833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63ffda2009a88f44b780b12064d25e88 with result from task 1833 parsl.dataflow.dflow DEBUG: Task 1833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0f0ea3f9-5cbf-45e5-bc46-ddb89711a85a_isrForCrosstalkSources_1031719_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/0f0ea3f9-5cbf-45e5-bc46-ddb89711a85a_isrForCrosstalkSources_1031719_6.stderr parsl.dataflow.dflow INFO: Task 1807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ada7acf3d4fb0005f1b508a5ec510032 with result from task 1807 parsl.dataflow.dflow DEBUG: Task 1807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/ae73726a-b134-496a-ada7-698c7cf6cbc8_isrForCrosstalkSources_1031710_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/ae73726a-b134-496a-ada7-698c7cf6cbc8_isrForCrosstalkSources_1031710_41.stderr parsl.dataflow.dflow INFO: Task 1918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e76a0b297ceacaf8ee905179e02df234 with result from task 1918 parsl.dataflow.dflow DEBUG: Task 1918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/eb01b66e-f3a5-48c6-a433-0c8489179263_isrForCrosstalkSources_1031723_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/eb01b66e-f3a5-48c6-a433-0c8489179263_isrForCrosstalkSources_1031723_25.stderr parsl.dataflow.dflow INFO: Task 2191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a520d38cf694e1642c6e69f9cbc17d7 with result from task 2191 parsl.dataflow.dflow DEBUG: Task 2191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/09cd52a6-c8fd-4fa5-bb7b-0b77cc0a9f4f_isrForCrosstalkSources_1031721_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/09cd52a6-c8fd-4fa5-bb7b-0b77cc0a9f4f_isrForCrosstalkSources_1031721_37.stderr parsl.dataflow.dflow INFO: Task 2193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bd058952064f0273aba1d336e8897ee with result from task 2193 parsl.dataflow.dflow DEBUG: Task 2193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a2109690-debd-4783-a7cb-99bdd07d803f_isrForCrosstalkSources_1031723_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/a2109690-debd-4783-a7cb-99bdd07d803f_isrForCrosstalkSources_1031723_24.stderr parsl.dataflow.dflow INFO: Task 2161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25b3551e7bec062386bf908e247ff897 with result from task 2161 parsl.dataflow.dflow DEBUG: Task 2161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/257d254d-acbd-4a6b-bfed-4a19fb9488d3_isrForCrosstalkSources_1031701_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/257d254d-acbd-4a6b-bfed-4a19fb9488d3_isrForCrosstalkSources_1031701_26.stderr parsl.dataflow.dflow INFO: Task 1914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c3dd105ad234afb43ef7ab37f369442 with result from task 1914 parsl.dataflow.dflow DEBUG: Task 1914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1657c898-aa0f-44b6-8c1f-e5f82730d2fd_isrForCrosstalkSources_1031700_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1657c898-aa0f-44b6-8c1f-e5f82730d2fd_isrForCrosstalkSources_1031700_2.stderr parsl.dataflow.dflow INFO: Task 2175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a32a3fa9ec79e03531831b1cc8f7b8db with result from task 2175 parsl.dataflow.dflow DEBUG: Task 2175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/fdb9408b-2415-4a78-823d-4cfe3f840859_isrForCrosstalkSources_1031724_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/fdb9408b-2415-4a78-823d-4cfe3f840859_isrForCrosstalkSources_1031724_25.stderr parsl.dataflow.dflow INFO: Task 1862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e1ca4d30a61f4e79f488f435f7cc0bd with result from task 1862 parsl.dataflow.dflow DEBUG: Task 1862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/51b3efd0-ab85-4662-8c1a-08c6df8104dc_isrForCrosstalkSources_1031729_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/51b3efd0-ab85-4662-8c1a-08c6df8104dc_isrForCrosstalkSources_1031729_38.stderr parsl.dataflow.dflow INFO: Task 2235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a9dc8d299d8038b97b20c8546505425 with result from task 2235 parsl.dataflow.dflow DEBUG: Task 2235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/6f669c9f-dc7f-4509-bbb8-8b4b221734a3_isrForCrosstalkSources_1031699_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/6f669c9f-dc7f-4509-bbb8-8b4b221734a3_isrForCrosstalkSources_1031699_37.stderr parsl.dataflow.dflow INFO: Task 1788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c6a37ca3a47b8ed89758e5d8dbd7498 with result from task 1788 parsl.dataflow.dflow DEBUG: Task 1788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/feba9287-5917-4627-bff2-5708f3312578_isrForCrosstalkSources_1031702_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/feba9287-5917-4627-bff2-5708f3312578_isrForCrosstalkSources_1031702_48.stderr parsl.dataflow.dflow INFO: Task 2212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5fd9999a78ffac4a99b2e379f94231f with result from task 2212 parsl.dataflow.dflow DEBUG: Task 2212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/33465f59-5aeb-4a81-aba2-a9ed2e1ac896_isrForCrosstalkSources_1031717_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/33465f59-5aeb-4a81-aba2-a9ed2e1ac896_isrForCrosstalkSources_1031717_32.stderr parsl.dataflow.dflow INFO: Task 2308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fdf6756414bbe6ac6fbc111c847da3b with result from task 2308 parsl.dataflow.dflow DEBUG: Task 2308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2a94c002-6195-4c43-9903-33005cfffbde_isrForCrosstalkSources_1031736_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/2a94c002-6195-4c43-9903-33005cfffbde_isrForCrosstalkSources_1031736_55.stderr parsl.dataflow.dflow INFO: Task 2178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddfaa8950028111a78c44b4376fa0af3 with result from task 2178 parsl.dataflow.dflow DEBUG: Task 2178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/611790c3-e9ae-426f-a212-9302e9fee8d0_isrForCrosstalkSources_1031705_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/611790c3-e9ae-426f-a212-9302e9fee8d0_isrForCrosstalkSources_1031705_40.stderr parsl.dataflow.dflow INFO: Task 1920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 759e481feb9431c38cdc885b39235b3a with result from task 1920 parsl.dataflow.dflow DEBUG: Task 1920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/3bc050d3-d8cc-4a13-a502-01749d99c82a_isrForCrosstalkSources_1031734_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/3bc050d3-d8cc-4a13-a502-01749d99c82a_isrForCrosstalkSources_1031734_56.stderr parsl.dataflow.dflow INFO: Task 2228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13aec746e47ce2555b92df9e7fab90c1 with result from task 2228 parsl.dataflow.dflow DEBUG: Task 2228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/9d7f78d5-1f6d-4e07-a1cc-cef704bd6c95_isrForCrosstalkSources_1031724_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/9d7f78d5-1f6d-4e07-a1cc-cef704bd6c95_isrForCrosstalkSources_1031724_34.stderr parsl.dataflow.dflow INFO: Task 2413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c383469f660702b492afb11a7c3cf000 with result from task 2413 parsl.dataflow.dflow DEBUG: Task 2413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/869f46ec-4c00-4887-b290-46394c7afad2_isrForCrosstalkSources_1031715_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/869f46ec-4c00-4887-b290-46394c7afad2_isrForCrosstalkSources_1031715_3.stderr parsl.dataflow.dflow INFO: Task 2171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d1d5b615806ed4946f5b7573b5663fc with result from task 2171 parsl.dataflow.dflow DEBUG: Task 2171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/cfc1dfca-0103-4f02-a855-589b06b72404_isrForCrosstalkSources_1031706_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/cfc1dfca-0103-4f02-a855-589b06b72404_isrForCrosstalkSources_1031706_34.stderr parsl.dataflow.dflow INFO: Task 2192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4485e5147c60b8aed41f803786bb0b6 with result from task 2192 parsl.dataflow.dflow DEBUG: Task 2192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/56bdbe1a-85b9-4e3f-b32b-3bf8c2aeaa70_isrForCrosstalkSources_1031723_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/56bdbe1a-85b9-4e3f-b32b-3bf8c2aeaa70_isrForCrosstalkSources_1031723_17.stderr parsl.dataflow.dflow INFO: Task 2009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f607b822f5ada6738f89a9a3c4756f9 with result from task 2009 parsl.dataflow.dflow DEBUG: Task 2009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/dd15db76-0d44-4f87-9521-6e270b7b5da2_isrForCrosstalkSources_1031709_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/dd15db76-0d44-4f87-9521-6e270b7b5da2_isrForCrosstalkSources_1031709_43.stderr parsl.dataflow.dflow INFO: Task 2014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9401afadf5133af71e9fa79b2ee7297d with result from task 2014 parsl.dataflow.dflow DEBUG: Task 2014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/01f8ce16-779e-4e74-93a8-a3c57409868b_isrForCrosstalkSources_1031722_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/01f8ce16-779e-4e74-93a8-a3c57409868b_isrForCrosstalkSources_1031722_32.stderr parsl.dataflow.dflow INFO: Task 1994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6308b03a31aa3e01531627dc7dc4a463 with result from task 1994 parsl.dataflow.dflow DEBUG: Task 1994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/bb2fe940-a386-4c28-926a-8f90f8c1b884_isrForCrosstalkSources_1031712_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/bb2fe940-a386-4c28-926a-8f90f8c1b884_isrForCrosstalkSources_1031712_34.stderr parsl.dataflow.dflow INFO: Task 2177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 313016bffcb9c48bb58e7d0bd7bdb87b with result from task 2177 parsl.dataflow.dflow DEBUG: Task 2177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c3e735e1-5c16-4f64-8efc-cea107f0c10b_isrForCrosstalkSources_1031716_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c3e735e1-5c16-4f64-8efc-cea107f0c10b_isrForCrosstalkSources_1031716_18.stderr parsl.dataflow.dflow INFO: Task 2096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c85acd8f2221cb5f44e7a80478f489fc with result from task 2096 parsl.dataflow.dflow DEBUG: Task 2096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/2e2ae47e-5ee8-4bd9-b8a2-333f7d307c69_isrForCrosstalkSources_1031725_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/2e2ae47e-5ee8-4bd9-b8a2-333f7d307c69_isrForCrosstalkSources_1031725_23.stderr parsl.dataflow.dflow INFO: Task 2073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68ccae5a917c80112effcb519129524e with result from task 2073 parsl.dataflow.dflow DEBUG: Task 2073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5c3efb72-ac68-447c-b2f5-e3e9e8aa96e4_isrForCrosstalkSources_1031724_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/5c3efb72-ac68-447c-b2f5-e3e9e8aa96e4_isrForCrosstalkSources_1031724_23.stderr parsl.dataflow.dflow INFO: Task 2187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e56116ca768ab2981cb75ba47f0196f with result from task 2187 parsl.dataflow.dflow DEBUG: Task 2187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/6e03efbc-1341-468f-a262-dc57d11407e0_isrForCrosstalkSources_1031706_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/6e03efbc-1341-468f-a262-dc57d11407e0_isrForCrosstalkSources_1031706_56.stderr parsl.dataflow.dflow INFO: Task 2302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffaea3e6c45d316feec731a8d41dc971 with result from task 2302 parsl.dataflow.dflow DEBUG: Task 2302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3b475d43-3300-4c13-9a79-a74d23b53d4e_isrForCrosstalkSources_1031723_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/3b475d43-3300-4c13-9a79-a74d23b53d4e_isrForCrosstalkSources_1031723_8.stderr parsl.dataflow.dflow INFO: Task 1851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d49ed8335ab680faa5b6905559c0920 with result from task 1851 parsl.dataflow.dflow DEBUG: Task 1851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/e0f0118d-2fcd-4703-acfc-9335aacd9f79_isrForCrosstalkSources_1031726_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/e0f0118d-2fcd-4703-acfc-9335aacd9f79_isrForCrosstalkSources_1031726_46.stderr parsl.dataflow.dflow INFO: Task 2150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10bc0e93787855a016aba612dc531d6f with result from task 2150 parsl.dataflow.dflow DEBUG: Task 2150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fa265ecd-0e58-4582-9089-9050ee6c2fc0_isrForCrosstalkSources_1031702_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fa265ecd-0e58-4582-9089-9050ee6c2fc0_isrForCrosstalkSources_1031702_19.stderr parsl.dataflow.dflow INFO: Task 2226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a42c49ea6ef8cb80b3bd7404b23185e5 with result from task 2226 parsl.dataflow.dflow DEBUG: Task 2226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/375e5568-f222-40f5-938c-d01b8284153a_isrForCrosstalkSources_1031731_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/375e5568-f222-40f5-938c-d01b8284153a_isrForCrosstalkSources_1031731_20.stderr parsl.dataflow.dflow INFO: Task 1858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a533626681d6582a9efaea9b288053c with result from task 1858 parsl.dataflow.dflow DEBUG: Task 1858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/0029f769-3b8a-44a8-a57c-861d834ff2a7_isrForCrosstalkSources_1031708_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/0029f769-3b8a-44a8-a57c-861d834ff2a7_isrForCrosstalkSources_1031708_61.stderr parsl.dataflow.dflow INFO: Task 1942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3381d205ef80b6f11cce4c5716178d85 with result from task 1942 parsl.dataflow.dflow DEBUG: Task 1942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/3039e25f-98c1-41f7-a315-eb5efd184952_isrForCrosstalkSources_1031733_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/3039e25f-98c1-41f7-a315-eb5efd184952_isrForCrosstalkSources_1031733_10.stderr parsl.dataflow.dflow INFO: Task 1774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a211da5da476e9fc53d708768e9a0e6 with result from task 1774 parsl.dataflow.dflow DEBUG: Task 1774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/fd75a39d-dc32-42e8-9334-1592441e1f35_isrForCrosstalkSources_1031735_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/fd75a39d-dc32-42e8-9334-1592441e1f35_isrForCrosstalkSources_1031735_62.stderr parsl.dataflow.dflow INFO: Task 1955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a333a5fc33699f81ea21d14b87b3fc84 with result from task 1955 parsl.dataflow.dflow DEBUG: Task 1955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c02374f4-42a6-4d3f-b6b1-dc8ba67d6f23_isrForCrosstalkSources_1031729_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/c02374f4-42a6-4d3f-b6b1-dc8ba67d6f23_isrForCrosstalkSources_1031729_48.stderr parsl.dataflow.dflow INFO: Task 2110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab62aa00b14247934ba6963b3cf4f7d5 with result from task 2110 parsl.dataflow.dflow DEBUG: Task 2110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/7050e05c-adaf-47cb-a8d7-b35edfe25cff_isrForCrosstalkSources_1031718_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/7050e05c-adaf-47cb-a8d7-b35edfe25cff_isrForCrosstalkSources_1031718_46.stderr parsl.dataflow.dflow INFO: Task 1821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c34e58631165cbfc22c5f594cdf54268 with result from task 1821 parsl.dataflow.dflow DEBUG: Task 1821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/36fdb4e2-9259-42fa-b052-87c25840ff85_isrForCrosstalkSources_1031724_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/36fdb4e2-9259-42fa-b052-87c25840ff85_isrForCrosstalkSources_1031724_57.stderr parsl.dataflow.dflow INFO: Task 2207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf2ff582c845c2c336bca6cdc8c2258f with result from task 2207 parsl.dataflow.dflow DEBUG: Task 2207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8e852631-baf2-4035-b33d-a18656b3fad1_isrForCrosstalkSources_1031721_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/8e852631-baf2-4035-b33d-a18656b3fad1_isrForCrosstalkSources_1031721_5.stderr parsl.dataflow.dflow INFO: Task 2349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e88b6d4f23d1c46c0af9e8d2556ee43d with result from task 2349 parsl.dataflow.dflow DEBUG: Task 2349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/6a853279-2f01-456b-a18e-8eb8806552f0_isrForCrosstalkSources_1031710_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/6a853279-2f01-456b-a18e-8eb8806552f0_isrForCrosstalkSources_1031710_23.stderr parsl.dataflow.dflow INFO: Task 2028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6066aaba36a4a41ae87b2e049b376d4e with result from task 2028 parsl.dataflow.dflow DEBUG: Task 2028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/633aec0c-844e-41e9-aabe-85f43458caf8_isrForCrosstalkSources_1031721_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/633aec0c-844e-41e9-aabe-85f43458caf8_isrForCrosstalkSources_1031721_41.stderr parsl.dataflow.dflow INFO: Task 1943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d73c7395f99d293ed76e1de7da48db07 with result from task 1943 parsl.dataflow.dflow DEBUG: Task 1943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2f6b004b-c3d4-46f5-9752-24a6fd18c16e_isrForCrosstalkSources_1031731_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2f6b004b-c3d4-46f5-9752-24a6fd18c16e_isrForCrosstalkSources_1031731_9.stderr parsl.dataflow.dflow INFO: Task 1888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ced045a7146ebb679f58d1536b649b4 with result from task 1888 parsl.dataflow.dflow DEBUG: Task 1888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2bcf4220-ae97-43cf-94b7-c4323acc79eb_isrForCrosstalkSources_1031703_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/2bcf4220-ae97-43cf-94b7-c4323acc79eb_isrForCrosstalkSources_1031703_50.stderr parsl.dataflow.dflow INFO: Task 2022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cb04ea4f524bbc23480e098d50b12ba with result from task 2022 parsl.dataflow.dflow DEBUG: Task 2022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/b038226f-240f-46a7-816b-f67c892824df_isrForCrosstalkSources_1031706_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/b038226f-240f-46a7-816b-f67c892824df_isrForCrosstalkSources_1031706_11.stderr parsl.dataflow.dflow INFO: Task 2211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a265f207ff8779b151abeb3e3b4124e2 with result from task 2211 parsl.dataflow.dflow DEBUG: Task 2211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d1e9afa1-cef9-4674-974b-0ebdb49968d4_isrForCrosstalkSources_1031715_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d1e9afa1-cef9-4674-974b-0ebdb49968d4_isrForCrosstalkSources_1031715_32.stderr parsl.dataflow.dflow INFO: Task 2236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d243fb657ae11da807d6749c683e42fd with result from task 2236 parsl.dataflow.dflow DEBUG: Task 2236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/26f82c2d-344a-4497-a3b0-aa0964d9d844_isrForCrosstalkSources_1031729_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/26f82c2d-344a-4497-a3b0-aa0964d9d844_isrForCrosstalkSources_1031729_35.stderr parsl.dataflow.dflow INFO: Task 1814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e851bdbe0ff4b5c61620843db4babc06 with result from task 1814 parsl.dataflow.dflow DEBUG: Task 1814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0237b4e9-a4f8-4840-b60b-d1953010b555_isrForCrosstalkSources_1031714_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0237b4e9-a4f8-4840-b60b-d1953010b555_isrForCrosstalkSources_1031714_57.stderr parsl.dataflow.dflow INFO: Task 2180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a3b93cd4d64b4590de005e208d76e4c with result from task 2180 parsl.dataflow.dflow DEBUG: Task 2180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a321d178-b0e1-4583-837f-c89a250a9026_isrForCrosstalkSources_1031706_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/a321d178-b0e1-4583-837f-c89a250a9026_isrForCrosstalkSources_1031706_5.stderr parsl.dataflow.dflow INFO: Task 2103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eabcbffe9bb0d07e394e5b798e55a415 with result from task 2103 parsl.dataflow.dflow DEBUG: Task 2103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/271966b6-8983-41e6-b306-ee0cef328c23_isrForCrosstalkSources_1031706_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/271966b6-8983-41e6-b306-ee0cef328c23_isrForCrosstalkSources_1031706_39.stderr parsl.dataflow.dflow INFO: Task 2390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7196bf292ebee146ee19c6e7a028c2bf with result from task 2390 parsl.dataflow.dflow DEBUG: Task 2390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/55df23ed-0eec-4f4b-8a8a-4ab8d0dc047b_isrForCrosstalkSources_1031702_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/55df23ed-0eec-4f4b-8a8a-4ab8d0dc047b_isrForCrosstalkSources_1031702_23.stderr parsl.dataflow.dflow INFO: Task 2065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e705e2169a9af089dcf941fcc574054e with result from task 2065 parsl.dataflow.dflow DEBUG: Task 2065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e9bd9524-1fd6-425b-8d28-b182bcada7ad_isrForCrosstalkSources_1031701_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e9bd9524-1fd6-425b-8d28-b182bcada7ad_isrForCrosstalkSources_1031701_6.stderr parsl.dataflow.dflow INFO: Task 1917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 029ebd05a3e8cad89b26506aa8564fe4 with result from task 1917 parsl.dataflow.dflow DEBUG: Task 1917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/53d8fb18-85c8-40a3-9f78-2b64cb70939a_isrForCrosstalkSources_1031730_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/53d8fb18-85c8-40a3-9f78-2b64cb70939a_isrForCrosstalkSources_1031730_30.stderr parsl.dataflow.dflow INFO: Task 2001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb25d8306394e4189cadacdfef74ed2b with result from task 2001 parsl.dataflow.dflow DEBUG: Task 2001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/147107b4-2966-45c6-8be7-daffad5e4bd0_isrForCrosstalkSources_1031720_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/147107b4-2966-45c6-8be7-daffad5e4bd0_isrForCrosstalkSources_1031720_7.stderr parsl.dataflow.dflow INFO: Task 1933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ead0cdfd0f9edd75e6a5f6e59dd401a with result from task 1933 parsl.dataflow.dflow DEBUG: Task 1933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/aa610763-9a0c-439e-9169-80a0d8cf1bea_isrForCrosstalkSources_1031710_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/aa610763-9a0c-439e-9169-80a0d8cf1bea_isrForCrosstalkSources_1031710_44.stderr parsl.dataflow.dflow INFO: Task 2140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 252d89d1bef58906eebc2369c3cc58eb with result from task 2140 parsl.dataflow.dflow DEBUG: Task 2140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/3716cb6d-ea99-46e7-95a1-3769c32d44ac_isrForCrosstalkSources_1031699_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/3716cb6d-ea99-46e7-95a1-3769c32d44ac_isrForCrosstalkSources_1031699_56.stderr parsl.dataflow.dflow INFO: Task 2075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6467d422d845fddef67e279e2e4d8963 with result from task 2075 parsl.dataflow.dflow DEBUG: Task 2075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/75189c41-6a89-41c6-b86a-5a543971cc94_isrForCrosstalkSources_1031738_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/75189c41-6a89-41c6-b86a-5a543971cc94_isrForCrosstalkSources_1031738_37.stderr parsl.dataflow.dflow INFO: Task 1905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 077078192b02ccb99981a71048782127 with result from task 1905 parsl.dataflow.dflow DEBUG: Task 1905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0f078bd4-a046-4fd9-bcd2-61cd85c5f9ba_isrForCrosstalkSources_1031714_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/0f078bd4-a046-4fd9-bcd2-61cd85c5f9ba_isrForCrosstalkSources_1031714_1.stderr parsl.dataflow.dflow INFO: Task 1952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e03c1123ac0c5f23b77117a7951730cf with result from task 1952 parsl.dataflow.dflow DEBUG: Task 1952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/0fd57141-ad72-4d4c-858a-4f49ceda2edd_isrForCrosstalkSources_1031701_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/0fd57141-ad72-4d4c-858a-4f49ceda2edd_isrForCrosstalkSources_1031701_5.stderr 2024-03-17 04:07:01 proc_lsst.multi:170 [INFO] [multi] job_id 17148182 proc_lsst.multi INFO: [multi] job_id 17148182 2024-03-17 04:07:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 179 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 179 2024-03-17 04:07:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148182 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148182 parsl.executors.status_handling DEBUG: Launched block 234 on executor multi with job ID 17148182 parsl.executors.status_handling INFO: Allocated block ID 235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-235 2024-03-17 04:07:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-235 2024-03-17 04:07:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 179 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 179 2024-03-17 04:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48b76349b2ddb9bfaf5ed6b8543f3225 with result from task 2435 parsl.dataflow.dflow DEBUG: Task 2435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/c286d528-78f0-4098-883b-34dde75f9a88_isrForCrosstalkSources_1031730_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/c286d528-78f0-4098-883b-34dde75f9a88_isrForCrosstalkSources_1031730_27.stderr parsl.dataflow.dflow INFO: Task 2137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34b677e7e4d422b221e374a8e15850db with result from task 2137 parsl.dataflow.dflow DEBUG: Task 2137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/e6bd5e9d-fec3-4019-a560-a267dc883ae4_isrForCrosstalkSources_1031699_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/e6bd5e9d-fec3-4019-a560-a267dc883ae4_isrForCrosstalkSources_1031699_12.stderr parsl.dataflow.dflow INFO: Task 2405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fd463fbe4e9a70a2ba134e8659caebc with result from task 2405 parsl.dataflow.dflow DEBUG: Task 2405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/4525466e-a8c4-4c26-8158-dc0524319dc0_isrForCrosstalkSources_1031702_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/4525466e-a8c4-4c26-8158-dc0524319dc0_isrForCrosstalkSources_1031702_29.stderr parsl.dataflow.dflow INFO: Task 2396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c995d700c91d970f9a9ee8da0931506 with result from task 2396 parsl.dataflow.dflow DEBUG: Task 2396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d3ebfbba-e37d-4d15-a46d-ed635cbd5616_isrForCrosstalkSources_1031722_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d3ebfbba-e37d-4d15-a46d-ed635cbd5616_isrForCrosstalkSources_1031722_29.stderr parsl.dataflow.dflow INFO: Task 2148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d28a1c7438712a96bc763ae60083efe9 with result from task 2148 parsl.dataflow.dflow DEBUG: Task 2148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/071d7eda-70c9-4a89-868a-dd16bbcdc7ba_isrForCrosstalkSources_1031729_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/071d7eda-70c9-4a89-868a-dd16bbcdc7ba_isrForCrosstalkSources_1031729_4.stderr parsl.dataflow.dflow INFO: Task 2411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5a6f92e0360e329b0583ffa2b91fcc3 with result from task 2411 parsl.dataflow.dflow DEBUG: Task 2411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/1b05e552-6409-41e5-a383-3acc4704afeb_isrForCrosstalkSources_1031726_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/1b05e552-6409-41e5-a383-3acc4704afeb_isrForCrosstalkSources_1031726_56.stderr parsl.dataflow.dflow INFO: Task 1997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa21c727fc48add56552d0051e7aeb47 with result from task 1997 parsl.dataflow.dflow DEBUG: Task 1997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/97cf1437-6dc4-432e-9165-f367aa53e068_isrForCrosstalkSources_1031736_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/97cf1437-6dc4-432e-9165-f367aa53e068_isrForCrosstalkSources_1031736_33.stderr parsl.dataflow.dflow INFO: Task 2017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f413af6665e8254feeeb91932c10d37f with result from task 2017 parsl.dataflow.dflow DEBUG: Task 2017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5e40ecb2-1c39-49b2-82ff-88d697d12c88_isrForCrosstalkSources_1031703_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/5e40ecb2-1c39-49b2-82ff-88d697d12c88_isrForCrosstalkSources_1031703_49.stderr parsl.dataflow.dflow INFO: Task 2195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db45d6e3e75a48d4a90606d058320e0e with result from task 2195 parsl.dataflow.dflow DEBUG: Task 2195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c2399dfa-14c9-4666-ba5c-dbf8f6876920_isrForCrosstalkSources_1031719_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c2399dfa-14c9-4666-ba5c-dbf8f6876920_isrForCrosstalkSources_1031719_28.stderr parsl.dataflow.dflow INFO: Task 1806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56346a5dff703aa167fc07ebc3ae842b with result from task 1806 parsl.dataflow.dflow DEBUG: Task 1806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/37b22a36-1878-4c02-98f7-c4513454f724_isrForCrosstalkSources_1031720_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/37b22a36-1878-4c02-98f7-c4513454f724_isrForCrosstalkSources_1031720_53.stderr parsl.dataflow.dflow INFO: Task 2287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4d01b11c51681f0ea11561d52e0e518 with result from task 2287 parsl.dataflow.dflow DEBUG: Task 2287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/5073facb-268f-45a6-9f84-143a36f7b074_isrForCrosstalkSources_1031702_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/5073facb-268f-45a6-9f84-143a36f7b074_isrForCrosstalkSources_1031702_24.stderr parsl.dataflow.dflow INFO: Task 2003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 540cdc78063202db29c14ffc88b450cc with result from task 2003 parsl.dataflow.dflow DEBUG: Task 2003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/2d004444-a75d-4ff3-83e4-608cb3f95a66_isrForCrosstalkSources_1031719_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/2d004444-a75d-4ff3-83e4-608cb3f95a66_isrForCrosstalkSources_1031719_39.stderr parsl.dataflow.dflow INFO: Task 2314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25d299fdd4a07fd6e7126b0460dfd1d7 with result from task 2314 parsl.dataflow.dflow DEBUG: Task 2314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b896192f-406a-4e27-bfdc-077362d78ca8_isrForCrosstalkSources_1031718_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/b896192f-406a-4e27-bfdc-077362d78ca8_isrForCrosstalkSources_1031718_51.stderr parsl.dataflow.dflow INFO: Task 2055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc2a50ed22e1223ea7f485bca31cea49 with result from task 2055 parsl.dataflow.dflow DEBUG: Task 2055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/14727c70-ac2f-4aa6-b098-97afcc0f8570_isrForCrosstalkSources_1031727_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/14727c70-ac2f-4aa6-b098-97afcc0f8570_isrForCrosstalkSources_1031727_26.stderr parsl.dataflow.dflow INFO: Task 2319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ac058a4ea6824ef164ed6111642edee with result from task 2319 parsl.dataflow.dflow DEBUG: Task 2319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/91ebfe14-9b52-4adb-ab28-bf6a5a75271d_isrForCrosstalkSources_1031699_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/91ebfe14-9b52-4adb-ab28-bf6a5a75271d_isrForCrosstalkSources_1031699_43.stderr parsl.dataflow.dflow INFO: Task 2072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 915a673067f90379957523c3a2c1cc0e with result from task 2072 parsl.dataflow.dflow DEBUG: Task 2072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/82d95df7-607f-46a3-894e-7b4a6929259c_isrForCrosstalkSources_1031733_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/82d95df7-607f-46a3-894e-7b4a6929259c_isrForCrosstalkSources_1031733_57.stderr parsl.dataflow.dflow INFO: Task 2299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ad800c4c0f904c516d83470520a4dbb with result from task 2299 parsl.dataflow.dflow DEBUG: Task 2299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/3931b165-a5b9-4f96-9222-d847bd5d812a_isrForCrosstalkSources_1031718_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/3931b165-a5b9-4f96-9222-d847bd5d812a_isrForCrosstalkSources_1031718_8.stderr parsl.dataflow.dflow INFO: Task 2056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a616e163f4181e3208dd0e02fda878e8 with result from task 2056 parsl.dataflow.dflow DEBUG: Task 2056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/fc821cc4-705e-449f-84d6-8ad84c07b570_isrForCrosstalkSources_1031726_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/fc821cc4-705e-449f-84d6-8ad84c07b570_isrForCrosstalkSources_1031726_5.stderr parsl.dataflow.dflow INFO: Task 2010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c4a3cc5b7394e5195c27b4e62383199 with result from task 2010 parsl.dataflow.dflow DEBUG: Task 2010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6c4f137a-d4ff-492f-8361-2031090f1085_isrForCrosstalkSources_1031721_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/6c4f137a-d4ff-492f-8361-2031090f1085_isrForCrosstalkSources_1031721_18.stderr parsl.dataflow.dflow INFO: Task 2071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfddf1c180c9200e91001ed7ff2ed4fe with result from task 2071 parsl.dataflow.dflow DEBUG: Task 2071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0850cce1-723b-40b8-b9ac-90c7c620f512_isrForCrosstalkSources_1031721_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/0850cce1-723b-40b8-b9ac-90c7c620f512_isrForCrosstalkSources_1031721_20.stderr parsl.dataflow.dflow INFO: Task 2082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73afe6450eceb8d149dbed63e6879103 with result from task 2082 parsl.dataflow.dflow DEBUG: Task 2082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/545e0216-6e76-419a-a86c-711751d92e27_isrForCrosstalkSources_1031709_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/545e0216-6e76-419a-a86c-711751d92e27_isrForCrosstalkSources_1031709_20.stderr parsl.dataflow.dflow INFO: Task 2074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9b4cc234a26bac049cb64699e3754cb with result from task 2074 parsl.dataflow.dflow DEBUG: Task 2074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/c43bdcae-efb6-483b-8c27-8c5c620f38d6_isrForCrosstalkSources_1031714_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/c43bdcae-efb6-483b-8c27-8c5c620f38d6_isrForCrosstalkSources_1031714_25.stderr parsl.dataflow.dflow INFO: Task 2415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27470d1176d203ffc65b49c6cb981bbd with result from task 2415 parsl.dataflow.dflow DEBUG: Task 2415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/c11cf64b-9c64-44ce-878d-b5189d6f788e_isrForCrosstalkSources_1031737_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/c11cf64b-9c64-44ce-878d-b5189d6f788e_isrForCrosstalkSources_1031737_20.stderr parsl.dataflow.dflow INFO: Task 2117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 684f3dcbc2a1ebf844b57d6305461d27 with result from task 2117 parsl.dataflow.dflow DEBUG: Task 2117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/d6d5df29-b354-48c6-ad76-ce783640552d_isrForCrosstalkSources_1031721_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/d6d5df29-b354-48c6-ad76-ce783640552d_isrForCrosstalkSources_1031721_21.stderr parsl.dataflow.dflow INFO: Task 2097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b910d6d0faf4d33b876d7edb9836a10 with result from task 2097 parsl.dataflow.dflow DEBUG: Task 2097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/10a6ce14-3a29-4b02-b249-0e5e4e90c32b_isrForCrosstalkSources_1031711_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/10a6ce14-3a29-4b02-b249-0e5e4e90c32b_isrForCrosstalkSources_1031711_19.stderr parsl.dataflow.dflow INFO: Task 2322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 287d8f90c8ff6ff37db8a84d5766907f with result from task 2322 parsl.dataflow.dflow DEBUG: Task 2322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e97e4c7d-4354-41c3-a81f-f21927836fd2_isrForCrosstalkSources_1031725_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e97e4c7d-4354-41c3-a81f-f21927836fd2_isrForCrosstalkSources_1031725_16.stderr parsl.dataflow.dflow INFO: Task 1999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1710616cfa82be81430d85426b084c45 with result from task 1999 parsl.dataflow.dflow DEBUG: Task 1999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/a64ab40d-a79f-4ce5-8bdf-54b987a0d2ae_isrForCrosstalkSources_1031700_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/a64ab40d-a79f-4ce5-8bdf-54b987a0d2ae_isrForCrosstalkSources_1031700_27.stderr parsl.dataflow.dflow INFO: Task 2197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebadad6979ded823dc1efa82a78ee0ce with result from task 2197 parsl.dataflow.dflow DEBUG: Task 2197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/4772ee43-a628-4ef0-91ef-859e477be4f4_isrForCrosstalkSources_1031731_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/4772ee43-a628-4ef0-91ef-859e477be4f4_isrForCrosstalkSources_1031731_59.stderr parsl.dataflow.dflow INFO: Task 1991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c96b6d34045de2a52adceabde9efbe40 with result from task 1991 parsl.dataflow.dflow DEBUG: Task 1991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/7ab70a61-b3ed-4b06-b652-0faf02fd419e_isrForCrosstalkSources_1031733_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/7ab70a61-b3ed-4b06-b652-0faf02fd419e_isrForCrosstalkSources_1031733_51.stderr parsl.dataflow.dflow INFO: Task 2286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1d287f90c63b600d1ec0082a0cbcb91 with result from task 2286 parsl.dataflow.dflow DEBUG: Task 2286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/918d01ea-740d-4e76-8819-6c590b8ce9e6_isrForCrosstalkSources_1031704_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/918d01ea-740d-4e76-8819-6c590b8ce9e6_isrForCrosstalkSources_1031704_22.stderr parsl.dataflow.dflow INFO: Task 1883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cbc6258d184ebf91cabc744c0986631 with result from task 1883 parsl.dataflow.dflow DEBUG: Task 1883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/df87e422-1f4a-4fdd-88d5-d30d8b98f930_isrForCrosstalkSources_1031732_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/df87e422-1f4a-4fdd-88d5-d30d8b98f930_isrForCrosstalkSources_1031732_15.stderr parsl.dataflow.dflow INFO: Task 2239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efc1a019b7419b07cac9d623673aa2d6 with result from task 2239 parsl.dataflow.dflow DEBUG: Task 2239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/afc94a00-4322-415c-a8e9-4449383a7208_isrForCrosstalkSources_1031726_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/afc94a00-4322-415c-a8e9-4449383a7208_isrForCrosstalkSources_1031726_42.stderr 2024-03-17 04:07:03 proc_lsst.multi:170 [INFO] [multi] job_id 17148183 proc_lsst.multi INFO: [multi] job_id 17148183 2024-03-17 04:07:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 180 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 180 2024-03-17 04:07:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148183 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148183 parsl.executors.status_handling DEBUG: Launched block 235 on executor multi with job ID 17148183 parsl.executors.status_handling INFO: Allocated block ID 236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-236 2024-03-17 04:07:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-236 2024-03-17 04:07:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 180 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 180 2024-03-17 04:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f67ed1ca02d8edf3f1bed6a244ce929 with result from task 1870 parsl.dataflow.dflow DEBUG: Task 1870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e9843898-5d9c-45b5-83bc-3ec76150ca32_isrForCrosstalkSources_1031736_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/e9843898-5d9c-45b5-83bc-3ec76150ca32_isrForCrosstalkSources_1031736_52.stderr parsl.dataflow.dflow INFO: Task 2345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67cd83e6de6be6f6df748df446f88c8b with result from task 2345 parsl.dataflow.dflow DEBUG: Task 2345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c2049d95-b2d7-4ca7-b614-173605e058ee_isrForCrosstalkSources_1031727_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/c2049d95-b2d7-4ca7-b614-173605e058ee_isrForCrosstalkSources_1031727_20.stderr parsl.dataflow.dflow INFO: Task 2076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d76bc6e52c8debb1c880990629f72a4f with result from task 2076 parsl.dataflow.dflow DEBUG: Task 2076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/9bc3bbea-3f20-49c4-a521-925f831c4ca4_isrForCrosstalkSources_1031708_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/9bc3bbea-3f20-49c4-a521-925f831c4ca4_isrForCrosstalkSources_1031708_51.stderr parsl.dataflow.dflow INFO: Task 2018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6979d1d5877ad9f30f9f36b3b1ca6ec with result from task 2018 parsl.dataflow.dflow DEBUG: Task 2018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cd4c79c9-c475-485c-a6a6-e37ee2def71f_isrForCrosstalkSources_1031710_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/cd4c79c9-c475-485c-a6a6-e37ee2def71f_isrForCrosstalkSources_1031710_48.stderr 2024-03-17 04:07:03 proc_lsst.multi:170 [INFO] [multi] job_id 17148184 proc_lsst.multi INFO: [multi] job_id 17148184 2024-03-17 04:07:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 181 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 181 2024-03-17 04:07:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148184 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148184 parsl.executors.status_handling DEBUG: Launched block 236 on executor multi with job ID 17148184 parsl.executors.status_handling INFO: Allocated block ID 237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-237 2024-03-17 04:07:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-237 2024-03-17 04:07:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 parsl.dataflow.dflow INFO: Task 2210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 032454ffb6dbb0333db0b73554736d12 with result from task 2210 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 181 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 181 2024-03-17 04:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 2210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7231d166-d8a5-4851-9878-ae8bf6895a02_isrForCrosstalkSources_1031714_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/7231d166-d8a5-4851-9878-ae8bf6895a02_isrForCrosstalkSources_1031714_8.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36f6e621df5ea3ea338078e77a200056 with result from task 1900 parsl.dataflow.dflow DEBUG: Task 1900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/93ab53b8-a812-4709-ae5b-2f779af5c012_isrForCrosstalkSources_1031737_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/93ab53b8-a812-4709-ae5b-2f779af5c012_isrForCrosstalkSources_1031737_35.stderr parsl.dataflow.dflow INFO: Task 2200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de797ccd09787b79a8bc3c56fc053447 with result from task 2200 parsl.dataflow.dflow DEBUG: Task 2200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/932a3e00-aa98-4f22-9a70-1e2d3443a355_isrForCrosstalkSources_1031706_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/932a3e00-aa98-4f22-9a70-1e2d3443a355_isrForCrosstalkSources_1031706_55.stderr parsl.dataflow.dflow INFO: Task 2330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b86ba0dba427a5850cf99a974ea617b with result from task 2330 parsl.dataflow.dflow DEBUG: Task 2330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a74ca528-ddc2-4773-982d-65bb1fcd4276_isrForCrosstalkSources_1031719_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/a74ca528-ddc2-4773-982d-65bb1fcd4276_isrForCrosstalkSources_1031719_25.stderr parsl.dataflow.dflow INFO: Task 2201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c2341356535039149e7aa8dbe8c5b1 with result from task 2201 parsl.dataflow.dflow DEBUG: Task 2201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/e23dd587-c20d-438a-85dd-ef690b6cc004_isrForCrosstalkSources_1031729_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/e23dd587-c20d-438a-85dd-ef690b6cc004_isrForCrosstalkSources_1031729_61.stderr parsl.dataflow.dflow INFO: Task 2206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf14a9a3c6981107fc3a4c4ed3a8fed2 with result from task 2206 parsl.dataflow.dflow DEBUG: Task 2206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/08ff2c75-ed1f-40ee-aa31-4c0b7173d528_isrForCrosstalkSources_1031728_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/08ff2c75-ed1f-40ee-aa31-4c0b7173d528_isrForCrosstalkSources_1031728_20.stderr parsl.dataflow.dflow INFO: Task 1805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cea816f6ee285261cbbae7117ef3af50 with result from task 1805 parsl.dataflow.dflow DEBUG: Task 1805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/147efcba-f90a-4e53-8086-65352cffa4a5_isrForCrosstalkSources_1031712_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/147efcba-f90a-4e53-8086-65352cffa4a5_isrForCrosstalkSources_1031712_50.stderr 2024-03-17 04:07:04 proc_lsst.multi:170 [INFO] [multi] job_id 17148185 proc_lsst.multi INFO: [multi] job_id 17148185 2024-03-17 04:07:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 182 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 182 2024-03-17 04:07:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148185 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148185 parsl.executors.status_handling DEBUG: Launched block 237 on executor multi with job ID 17148185 parsl.executors.status_handling INFO: Allocated block ID 238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-238 2024-03-17 04:07:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-238 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 182 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 182 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d4446cc431298f7af7865516e6bf985 with result from task 1789 parsl.dataflow.dflow DEBUG: Task 1789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/bed12a83-fe40-4c05-87f1-686f89ceb69c_isrForCrosstalkSources_1031718_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/bed12a83-fe40-4c05-87f1-686f89ceb69c_isrForCrosstalkSources_1031718_36.stderr parsl.dataflow.dflow INFO: Task 2242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e76cc1d0dc18ff5596ae2e3920bf579 with result from task 2242 parsl.dataflow.dflow DEBUG: Task 2242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9f261731-65a1-4bea-ba34-c4286bfbfda8_isrForCrosstalkSources_1031716_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/9f261731-65a1-4bea-ba34-c4286bfbfda8_isrForCrosstalkSources_1031716_2.stderr parsl.dataflow.dflow INFO: Task 2196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00c15b21d0af7d4e6a0a931e1128992c with result from task 2196 parsl.dataflow.dflow DEBUG: Task 2196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/35aededa-88b0-4be7-b1dd-f83f2b0b5d3f_isrForCrosstalkSources_1031701_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/35aededa-88b0-4be7-b1dd-f83f2b0b5d3f_isrForCrosstalkSources_1031701_59.stderr parsl.dataflow.dflow INFO: Task 2027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 048e5f6561bcab6969188ae7fab7cd9d with result from task 2027 parsl.dataflow.dflow DEBUG: Task 2027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/b715adc2-cc09-4fe5-94e4-213097143b00_isrForCrosstalkSources_1031738_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/b715adc2-cc09-4fe5-94e4-213097143b00_isrForCrosstalkSources_1031738_41.stderr parsl.dataflow.dflow INFO: Task 2381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db0c505606f312ce668d7e4087c354f8 with result from task 2381 parsl.dataflow.dflow DEBUG: Task 2381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/9b62fbd3-e390-4ddf-a0f0-6ba6cd4a6fb0_isrForCrosstalkSources_1031713_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/9b62fbd3-e390-4ddf-a0f0-6ba6cd4a6fb0_isrForCrosstalkSources_1031713_19.stderr parsl.dataflow.dflow INFO: Task 2402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aaaccc65a7368cbb6ef33dc5e4ba0fb with result from task 2402 parsl.dataflow.dflow DEBUG: Task 2402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d281c170-51ea-4707-9453-e4e4d95e268f_isrForCrosstalkSources_1031730_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d281c170-51ea-4707-9453-e4e4d95e268f_isrForCrosstalkSources_1031730_40.stderr parsl.dataflow.dflow INFO: Task 2198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f126a27cfabdf6228c484c789255b509 with result from task 2198 parsl.dataflow.dflow DEBUG: Task 2198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/e1c2f7eb-2291-4449-b2c7-c857cb505a06_isrForCrosstalkSources_1031734_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/e1c2f7eb-2291-4449-b2c7-c857cb505a06_isrForCrosstalkSources_1031734_36.stderr 2024-03-17 04:07:04 proc_lsst.multi:170 [INFO] [multi] job_id 17148186 parsl.dataflow.dflow INFO: Task 1989 completed (launched -> exec_done) proc_lsst.multi INFO: [multi] job_id 17148186 2024-03-17 04:07:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 183 parsl.dataflow.memoization DEBUG: Storing app cache entry bfb0e942311d5a4a0b869e629529ba9a with result from task 1989 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 183 2024-03-17 04:07:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148186 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148186 parsl.executors.status_handling DEBUG: Launched block 238 on executor multi with job ID 17148186 parsl.dataflow.dflow DEBUG: Task 1989 checkpointed parsl.executors.status_handling INFO: Allocated block ID 239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-239 2024-03-17 04:07:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-239 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-239 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 parsl.dataflow.dflow INFO: Standard output for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ed81e890-15f0-46ca-b604-2d7ac75d835f_isrForCrosstalkSources_1031713_8.stdout proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 parsl.dataflow.dflow INFO: Standard error for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/ed81e890-15f0-46ca-b604-2d7ac75d835f_isrForCrosstalkSources_1031713_8.stderr proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 parsl.dataflow.dflow INFO: Task 2377 completed (launched -> exec_done) proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 parsl.dataflow.memoization DEBUG: Storing app cache entry d1a3ed06df706ac859f2668dd799e2f5 with result from task 2377 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 parsl.dataflow.dflow DEBUG: Task 2377 checkpointed proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Standard output for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/eaa1549e-20ba-4a6c-ab08-4343e15e2b20_isrForCrosstalkSources_1031726_34.stdout proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 183 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 183 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.dflow INFO: Standard error for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/eaa1549e-20ba-4a6c-ab08-4343e15e2b20_isrForCrosstalkSources_1031726_34.stderr parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d22d76195aaabb1b7054ea8123933c7 with result from task 2101 parsl.dataflow.dflow DEBUG: Task 2101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/04af23cd-7287-49b5-9f86-670da5ededb1_isrForCrosstalkSources_1031720_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/04af23cd-7287-49b5-9f86-670da5ededb1_isrForCrosstalkSources_1031720_28.stderr parsl.dataflow.dflow INFO: Task 2213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f1b86c8c8b2e71debac463665e6edaf with result from task 2213 parsl.dataflow.dflow DEBUG: Task 2213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/218fa0eb-9481-458c-a629-eafa6c329047_isrForCrosstalkSources_1031738_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/218fa0eb-9481-458c-a629-eafa6c329047_isrForCrosstalkSources_1031738_18.stderr parsl.dataflow.dflow INFO: Task 2430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 966edba09646ee6c6bff128e4aa05639 with result from task 2430 parsl.dataflow.dflow DEBUG: Task 2430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/2e2dee64-b2c2-4371-aae1-ac953330a3d1_isrForCrosstalkSources_1031737_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/2e2dee64-b2c2-4371-aae1-ac953330a3d1_isrForCrosstalkSources_1031737_21.stderr parsl.dataflow.dflow INFO: Task 2409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d81e1644b7345ce5a430fff37029c06a with result from task 2409 parsl.dataflow.dflow DEBUG: Task 2409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/9efcf06b-ed2f-4c1b-9e95-bdfafe729e0c_isrForCrosstalkSources_1031700_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/9efcf06b-ed2f-4c1b-9e95-bdfafe729e0c_isrForCrosstalkSources_1031700_16.stderr parsl.dataflow.dflow INFO: Task 2199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ca26a75c410b6b7196ea8c6e7c0c25a with result from task 2199 parsl.dataflow.dflow DEBUG: Task 2199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/514dcb61-c15a-4961-af5c-0f7669f8716c_isrForCrosstalkSources_1031718_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/514dcb61-c15a-4961-af5c-0f7669f8716c_isrForCrosstalkSources_1031718_44.stderr 2024-03-17 04:07:04 proc_lsst.multi:170 [INFO] [multi] job_id 17148187 proc_lsst.multi INFO: [multi] job_id 17148187 2024-03-17 04:07:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 184 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 184 2024-03-17 04:07:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148187 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148187 parsl.executors.status_handling DEBUG: Launched block 239 on executor multi with job ID 17148187 parsl.executors.status_handling INFO: Allocated block ID 240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-240 2024-03-17 04:07:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-240 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 184 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 184 2024-03-17 04:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 599508eb53a19a5547a9b1046c3c62bb with result from task 2092 parsl.dataflow.dflow DEBUG: Task 2092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6eb8f699-1b6c-47c0-ba01-0bba9b0fdd16_isrForCrosstalkSources_1031700_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/6eb8f699-1b6c-47c0-ba01-0bba9b0fdd16_isrForCrosstalkSources_1031700_56.stderr parsl.dataflow.dflow INFO: Task 2146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f0ee969a1b9c1ade8147716b0e2fafd with result from task 2146 parsl.dataflow.dflow DEBUG: Task 2146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/49c9e811-ce4c-49c7-af20-98b27f304db3_isrForCrosstalkSources_1031730_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/49c9e811-ce4c-49c7-af20-98b27f304db3_isrForCrosstalkSources_1031730_21.stderr parsl.dataflow.dflow INFO: Task 1954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fb1204515559902af6d8e61cfda6525 with result from task 1954 parsl.dataflow.dflow DEBUG: Task 1954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/5168fc8a-dead-43ef-9709-29aaeddfd9bb_isrForCrosstalkSources_1031713_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/5168fc8a-dead-43ef-9709-29aaeddfd9bb_isrForCrosstalkSources_1031713_17.stderr parsl.dataflow.dflow INFO: Task 2194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f093bf5afce5a80a6067ed211f44dd7 with result from task 2194 parsl.dataflow.dflow DEBUG: Task 2194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/5d85f309-5c4c-4efc-a3a2-be1caba76488_isrForCrosstalkSources_1031733_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/5d85f309-5c4c-4efc-a3a2-be1caba76488_isrForCrosstalkSources_1031733_42.stderr parsl.dataflow.dflow INFO: Task 2387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79632fc7faa59d2cd89f14304d73e751 with result from task 2387 parsl.dataflow.dflow DEBUG: Task 2387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b8cba738-2e19-4d30-a596-d4b09936f52d_isrForCrosstalkSources_1031716_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/b8cba738-2e19-4d30-a596-d4b09936f52d_isrForCrosstalkSources_1031716_4.stderr parsl.dataflow.dflow INFO: Task 2477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 787c77d481b318f774c43e69c8471adf with result from task 2477 parsl.dataflow.dflow DEBUG: Task 2477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/05adaa7a-2c5e-42ac-aa08-04c10c42aa8a_isrForCrosstalkSources_1031736_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/05adaa7a-2c5e-42ac-aa08-04c10c42aa8a_isrForCrosstalkSources_1031736_27.stderr parsl.dataflow.dflow INFO: Task 2392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5415a153630fa6ad2d31c4525106899f with result from task 2392 parsl.dataflow.dflow DEBUG: Task 2392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c57fd8ee-123a-47be-87c9-b12c936a20cb_isrForCrosstalkSources_1031734_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c57fd8ee-123a-47be-87c9-b12c936a20cb_isrForCrosstalkSources_1031734_15.stderr 2024-03-17 04:07:05 proc_lsst.multi:170 [INFO] [multi] job_id 17148188 proc_lsst.multi INFO: [multi] job_id 17148188 2024-03-17 04:07:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 185 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 185 2024-03-17 04:07:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148188 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148188 parsl.executors.status_handling DEBUG: Launched block 240 on executor multi with job ID 17148188 parsl.executors.status_handling INFO: Allocated block ID 241 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-241 2024-03-17 04:07:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-241 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-241 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 185 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 185 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2f0e1bcc058fa39e4cb66b51120b0cb with result from task 1876 parsl.dataflow.dflow DEBUG: Task 1876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/34547987-5674-4473-8848-ab4459a9748f_isrForCrosstalkSources_1031704_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/34547987-5674-4473-8848-ab4459a9748f_isrForCrosstalkSources_1031704_59.stderr parsl.dataflow.dflow INFO: Task 1860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3d9c0bce6c6eacdbef266e2247a4489 with result from task 1860 parsl.dataflow.dflow DEBUG: Task 1860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a74db3b1-23bf-4dea-8b12-41a3a2a94877_isrForCrosstalkSources_1031702_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a74db3b1-23bf-4dea-8b12-41a3a2a94877_isrForCrosstalkSources_1031702_49.stderr 2024-03-17 04:07:05 proc_lsst.multi:170 [INFO] [multi] job_id 17148189 proc_lsst.multi INFO: [multi] job_id 17148189 2024-03-17 04:07:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 186 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 186 2024-03-17 04:07:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148189 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148189 parsl.executors.status_handling DEBUG: Launched block 241 on executor multi with job ID 17148189 parsl.executors.status_handling INFO: Allocated block ID 242 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-242 2024-03-17 04:07:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-242 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-242 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 186 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 186 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f9467f2f10319b55522f162647672f1 with result from task 2205 parsl.dataflow.dflow DEBUG: Task 2205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/59d75d27-0bf1-40af-91e7-13374e59d709_isrForCrosstalkSources_1031699_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/59d75d27-0bf1-40af-91e7-13374e59d709_isrForCrosstalkSources_1031699_58.stderr parsl.dataflow.dflow INFO: Task 1880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 476cccf4e6e22aeb16ab3017671311b9 with result from task 1880 parsl.dataflow.dflow DEBUG: Task 1880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/1658c3c6-4148-4edf-b9c0-75aa157ba141_isrForCrosstalkSources_1031724_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/1658c3c6-4148-4edf-b9c0-75aa157ba141_isrForCrosstalkSources_1031724_24.stderr parsl.dataflow.dflow INFO: Task 2183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96504846fc4f07c10e2ab0c331832830 with result from task 2183 parsl.dataflow.dflow DEBUG: Task 2183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/862296ea-1762-4fb6-88ae-dfa23ef79e9f_isrForCrosstalkSources_1031716_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/862296ea-1762-4fb6-88ae-dfa23ef79e9f_isrForCrosstalkSources_1031716_54.stderr parsl.dataflow.dflow INFO: Task 1945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 232e0b21ab4834950c9f0caf7fe282d4 with result from task 1945 parsl.dataflow.dflow DEBUG: Task 1945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/97265be7-e3c6-45a1-a50a-f131fdab0c00_isrForCrosstalkSources_1031731_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/97265be7-e3c6-45a1-a50a-f131fdab0c00_isrForCrosstalkSources_1031731_44.stderr parsl.dataflow.dflow INFO: Task 2168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d74f8eeee4f216a1bc895bf8b3ae3bcb with result from task 2168 parsl.dataflow.dflow DEBUG: Task 2168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/55d2a2c8-5092-4243-8f7d-24ba30ec6a16_isrForCrosstalkSources_1031731_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/55d2a2c8-5092-4243-8f7d-24ba30ec6a16_isrForCrosstalkSources_1031731_2.stderr parsl.dataflow.dflow INFO: Task 2449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d481e69bd56659259ec05db76790d6c with result from task 2449 2024-03-17 04:07:05 proc_lsst.multi:170 [INFO] [multi] job_id 17148190 parsl.dataflow.dflow DEBUG: Task 2449 checkpointed proc_lsst.multi INFO: [multi] job_id 17148190 2024-03-17 04:07:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 187 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 187 2024-03-17 04:07:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148190 parsl.dataflow.dflow INFO: Standard output for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/5a9f9c25-d1fa-480b-ba31-eb9da287bf21_isrForCrosstalkSources_1031725_48.stdout proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148190 parsl.dataflow.dflow INFO: Standard error for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/5a9f9c25-d1fa-480b-ba31-eb9da287bf21_isrForCrosstalkSources_1031725_48.stderr parsl.executors.status_handling DEBUG: Launched block 242 on executor multi with job ID 17148190 parsl.executors.status_handling INFO: Allocated block ID 243 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-243 2024-03-17 04:07:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-243 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-243 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 187 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 187 2024-03-17 04:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dd85c640e26b9c6f64c5bd0005dbf69 with result from task 2465 parsl.dataflow.dflow DEBUG: Task 2465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/866ccc3a-8dbb-4f87-82d8-4b8c2742436f_isrForCrosstalkSources_1031725_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/866ccc3a-8dbb-4f87-82d8-4b8c2742436f_isrForCrosstalkSources_1031725_45.stderr parsl.dataflow.dflow INFO: Task 2068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fe6c8a4241deb1983151e230597f507 with result from task 2068 parsl.dataflow.dflow DEBUG: Task 2068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/fe2ed599-7e1f-400d-95d1-3c22a033da19_isrForCrosstalkSources_1031709_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/fe2ed599-7e1f-400d-95d1-3c22a033da19_isrForCrosstalkSources_1031709_32.stderr parsl.dataflow.dflow INFO: Task 2220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f53d0c57667199cc408417de48b55de with result from task 2220 parsl.dataflow.dflow DEBUG: Task 2220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3b6c8243-21da-40fa-a405-468061d354e6_isrForCrosstalkSources_1031713_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/3b6c8243-21da-40fa-a405-468061d354e6_isrForCrosstalkSources_1031713_53.stderr parsl.dataflow.dflow INFO: Task 1985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7aaf36732cf8a5050e6a84d343dc311 with result from task 1985 parsl.dataflow.dflow DEBUG: Task 1985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/9dfe3a6c-8bbf-4bdf-bcf4-449de721dabd_isrForCrosstalkSources_1031712_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/9dfe3a6c-8bbf-4bdf-bcf4-449de721dabd_isrForCrosstalkSources_1031712_14.stderr parsl.dataflow.dflow INFO: Task 2115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a01572a3921245826863bd8bd6d9b05 with result from task 2115 parsl.dataflow.dflow DEBUG: Task 2115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/66462ffe-5330-45fa-b744-f679be70eb9c_isrForCrosstalkSources_1031729_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/66462ffe-5330-45fa-b744-f679be70eb9c_isrForCrosstalkSources_1031729_14.stderr parsl.dataflow.dflow INFO: Task 2162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94d25d23df7bb82dbb9fdbc7b7346d14 with result from task 2162 parsl.dataflow.dflow DEBUG: Task 2162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/3d56dc7d-44a5-4aac-9c21-a1bc4650e6ca_isrForCrosstalkSources_1031737_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/3d56dc7d-44a5-4aac-9c21-a1bc4650e6ca_isrForCrosstalkSources_1031737_40.stderr 2024-03-17 04:07:06 proc_lsst.multi:170 [INFO] [multi] job_id 17148191 proc_lsst.multi INFO: [multi] job_id 17148191 2024-03-17 04:07:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 188 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 188 2024-03-17 04:07:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148191 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148191 parsl.executors.status_handling DEBUG: Launched block 243 on executor multi with job ID 17148191 parsl.executors.status_handling INFO: Allocated block ID 244 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-244 2024-03-17 04:07:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-244 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-244 2024-03-17 04:07:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 188 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 188 2024-03-17 04:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3db489bce85869eeab24435feeedaaa with result from task 1916 parsl.dataflow.dflow DEBUG: Task 1916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/28129b60-160e-4f10-bdd5-ebcd9cb5c02c_isrForCrosstalkSources_1031702_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/28129b60-160e-4f10-bdd5-ebcd9cb5c02c_isrForCrosstalkSources_1031702_62.stderr parsl.dataflow.dflow INFO: Task 2107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e820c2240c0e2cdfb23b360cc84f892b with result from task 2107 parsl.dataflow.dflow DEBUG: Task 2107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/0512db29-23db-42ea-a79c-99cde9de641d_isrForCrosstalkSources_1031729_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/0512db29-23db-42ea-a79c-99cde9de641d_isrForCrosstalkSources_1031729_43.stderr parsl.dataflow.dflow INFO: Task 2007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b068c5c6d5b58a03d6b4a68eac553278 with result from task 2007 parsl.dataflow.dflow DEBUG: Task 2007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/87ad27dd-5174-4828-a61d-ca47e231b8ae_isrForCrosstalkSources_1031727_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/87ad27dd-5174-4828-a61d-ca47e231b8ae_isrForCrosstalkSources_1031727_58.stderr parsl.dataflow.dflow INFO: Task 2166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f76b4af25d69ac1ea63de156b83a1215 with result from task 2166 parsl.dataflow.dflow DEBUG: Task 2166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d9e38d54-f535-4223-83b4-2c0aaf4c0af3_isrForCrosstalkSources_1031738_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/d9e38d54-f535-4223-83b4-2c0aaf4c0af3_isrForCrosstalkSources_1031738_20.stderr 2024-03-17 04:07:07 proc_lsst.multi:170 [INFO] [multi] job_id 17148192 proc_lsst.multi INFO: [multi] job_id 17148192 2024-03-17 04:07:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 189 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 189 2024-03-17 04:07:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148192 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148192 parsl.executors.status_handling DEBUG: Launched block 244 on executor multi with job ID 17148192 parsl.executors.status_handling INFO: Allocated block ID 245 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-245 2024-03-17 04:07:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-245 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-245 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 189 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 189 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ddce6e7aaef3a20c0b74cd204c403f2 with result from task 1947 parsl.dataflow.dflow DEBUG: Task 1947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/7cc8d542-13de-43e9-a636-1cccb69839d0_isrForCrosstalkSources_1031717_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/7cc8d542-13de-43e9-a636-1cccb69839d0_isrForCrosstalkSources_1031717_60.stderr parsl.dataflow.dflow INFO: Task 2141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1a3ddfdfe053506fc854367acedabec with result from task 2141 parsl.dataflow.dflow DEBUG: Task 2141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/c424b95e-3404-4b01-b6a3-ade2c10483b9_isrForCrosstalkSources_1031704_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/c424b95e-3404-4b01-b6a3-ade2c10483b9_isrForCrosstalkSources_1031704_37.stderr parsl.dataflow.dflow INFO: Task 1925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0b8cf09192bd4d75c6ddafbfdd904df with result from task 1925 parsl.dataflow.dflow DEBUG: Task 1925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/05fc06bb-94a3-4907-8e3f-3aafdfd590ad_isrForCrosstalkSources_1031727_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/05fc06bb-94a3-4907-8e3f-3aafdfd590ad_isrForCrosstalkSources_1031727_15.stderr parsl.dataflow.dflow INFO: Task 2327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2440fe41db591f919eb7c2a1fa933fdf with result from task 2327 parsl.dataflow.dflow DEBUG: Task 2327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0cfff05e-c31d-4780-b1f6-65fae78a022c_isrForCrosstalkSources_1031712_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/0cfff05e-c31d-4780-b1f6-65fae78a022c_isrForCrosstalkSources_1031712_20.stderr parsl.dataflow.dflow INFO: Task 2124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a51d27cda18bab14c91c8de8b938ea9 with result from task 2124 parsl.dataflow.dflow DEBUG: Task 2124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/b1d64d71-fcaa-451f-a8c0-b42bab019285_isrForCrosstalkSources_1031709_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/b1d64d71-fcaa-451f-a8c0-b42bab019285_isrForCrosstalkSources_1031709_45.stderr parsl.dataflow.dflow INFO: Task 2019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6095326087f7b228370d59bd1cc4d82 with result from task 2019 parsl.dataflow.dflow DEBUG: Task 2019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/047a2ece-a10b-4890-ac4e-9e57d77675b3_isrForCrosstalkSources_1031736_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/047a2ece-a10b-4890-ac4e-9e57d77675b3_isrForCrosstalkSources_1031736_49.stderr parsl.dataflow.dflow INFO: Task 2447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76867cdc921ff3c43996fba126c4628c with result from task 2447 parsl.dataflow.dflow DEBUG: Task 2447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/56fe5671-9ea8-4a6b-ba5d-d5e16b03ec1e_isrForCrosstalkSources_1031710_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/56fe5671-9ea8-4a6b-ba5d-d5e16b03ec1e_isrForCrosstalkSources_1031710_19.stderr parsl.dataflow.dflow INFO: Task 1886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5edd4e7cb990ee763b06265acd9de64e with result from task 1886 parsl.dataflow.dflow DEBUG: Task 1886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fc889e5f-124f-418f-8372-9230b29b0ba9_isrForCrosstalkSources_1031729_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/fc889e5f-124f-418f-8372-9230b29b0ba9_isrForCrosstalkSources_1031729_10.stderr parsl.dataflow.dflow INFO: Task 2156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abf6cbe1a6a293dd043af10577d40c1d with result from task 2156 parsl.dataflow.dflow DEBUG: Task 2156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1a5e23cd-f8cb-4d35-bec8-4d4b933d7277_isrForCrosstalkSources_1031733_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/1a5e23cd-f8cb-4d35-bec8-4d4b933d7277_isrForCrosstalkSources_1031733_45.stderr 2024-03-17 04:07:07 proc_lsst.multi:170 [INFO] [multi] job_id 17148193 proc_lsst.multi INFO: [multi] job_id 17148193 2024-03-17 04:07:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 190 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 190 2024-03-17 04:07:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148193 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148193 parsl.executors.status_handling DEBUG: Launched block 245 on executor multi with job ID 17148193 parsl.executors.status_handling INFO: Allocated block ID 246 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-246 2024-03-17 04:07:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-246 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-246 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 190 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 190 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry deac9a0acd1119f3575fdb6b3c695d73 with result from task 2326 parsl.dataflow.dflow DEBUG: Task 2326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/87e9875b-ced8-4266-89fb-6f4ba77d0add_isrForCrosstalkSources_1031715_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/87e9875b-ced8-4266-89fb-6f4ba77d0add_isrForCrosstalkSources_1031715_16.stderr parsl.dataflow.dflow INFO: Task 2389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe5c14beeef8a0c19c42f89aa826194a with result from task 2389 parsl.dataflow.dflow DEBUG: Task 2389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/fcf80a3c-727a-473b-b84e-25b78b95a4ca_isrForCrosstalkSources_1031718_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/fcf80a3c-727a-473b-b84e-25b78b95a4ca_isrForCrosstalkSources_1031718_29.stderr parsl.dataflow.dflow INFO: Task 2257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38269d6f49379d587b6517de2d5cb421 with result from task 2257 parsl.dataflow.dflow DEBUG: Task 2257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/773f1962-1ad9-4b8f-943a-3b6152d740c6_isrForCrosstalkSources_1031725_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/773f1962-1ad9-4b8f-943a-3b6152d740c6_isrForCrosstalkSources_1031725_27.stderr parsl.dataflow.dflow INFO: Task 2164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a802079ba4b1778c0c3f0d58be9d1720 with result from task 2164 parsl.dataflow.dflow DEBUG: Task 2164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8c91f415-5d88-4c6a-98da-3457a63518c9_isrForCrosstalkSources_1031707_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8c91f415-5d88-4c6a-98da-3457a63518c9_isrForCrosstalkSources_1031707_22.stderr parsl.dataflow.dflow INFO: Task 2222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7c4ae7681d0f6c275085de06331e5b6 with result from task 2222 parsl.dataflow.dflow DEBUG: Task 2222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/506feb06-cfc1-4b42-8ffd-f70924c202a4_isrForCrosstalkSources_1031723_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/506feb06-cfc1-4b42-8ffd-f70924c202a4_isrForCrosstalkSources_1031723_51.stderr parsl.dataflow.dflow INFO: Task 2024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e95aa48a9b467f64b8b993ebb33e1444 with result from task 2024 parsl.dataflow.dflow DEBUG: Task 2024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f5c5580e-d06a-48a4-b0c0-d8e784f7f4f0_isrForCrosstalkSources_1031721_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f5c5580e-d06a-48a4-b0c0-d8e784f7f4f0_isrForCrosstalkSources_1031721_56.stderr parsl.dataflow.dflow INFO: Task 2258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00a059f30f2cec20469935b5fc0c425a with result from task 2258 parsl.dataflow.dflow DEBUG: Task 2258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b584f179-1909-40b1-8bc8-2df4692617aa_isrForCrosstalkSources_1031707_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/b584f179-1909-40b1-8bc8-2df4692617aa_isrForCrosstalkSources_1031707_33.stderr parsl.dataflow.dflow INFO: Task 2238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 471d1d4ca938afcf3873c556e29f5d97 with result from task 2238 parsl.dataflow.dflow DEBUG: Task 2238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2fb1cdae-6a87-40a4-884c-4303413c77e2_isrForCrosstalkSources_1031723_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2fb1cdae-6a87-40a4-884c-4303413c77e2_isrForCrosstalkSources_1031723_41.stderr parsl.dataflow.dflow INFO: Task 2155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06ba796208bcedc7762275c57799c5fe with result from task 2155 parsl.dataflow.dflow DEBUG: Task 2155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/379fb331-a237-4045-a256-f0a88e35d9ef_isrForCrosstalkSources_1031733_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/379fb331-a237-4045-a256-f0a88e35d9ef_isrForCrosstalkSources_1031733_18.stderr 2024-03-17 04:07:07 proc_lsst.multi:170 [INFO] [multi] job_id 17148194 proc_lsst.multi INFO: [multi] job_id 17148194 2024-03-17 04:07:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 191 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 191 2024-03-17 04:07:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148194 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148194 parsl.executors.status_handling DEBUG: Launched block 246 on executor multi with job ID 17148194 parsl.dataflow.dflow INFO: Task 2167 completed (launched -> exec_done) parsl.executors.status_handling INFO: Allocated block ID 247 parsl.dataflow.memoization DEBUG: Storing app cache entry 00bfe3f3910bcaec7ff09b015f789364 with result from task 2167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-247 2024-03-17 04:07:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-247 parsl.dataflow.dflow DEBUG: Task 2167 checkpointed proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-247 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 parsl.dataflow.dflow INFO: Standard output for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ae7323b4-5495-4619-be04-575fd11a2b4e_isrForCrosstalkSources_1031699_19.stdout proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Standard error for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/ae7323b4-5495-4619-be04-575fd11a2b4e_isrForCrosstalkSources_1031699_19.stderr proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 191 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 191 2024-03-17 04:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f51be246abcf4f8be09e7a498fab0c6 with result from task 2152 parsl.dataflow.dflow DEBUG: Task 2152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/15a99062-99bc-4982-b39b-105df4d0effc_isrForCrosstalkSources_1031731_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/15a99062-99bc-4982-b39b-105df4d0effc_isrForCrosstalkSources_1031731_52.stderr parsl.dataflow.dflow INFO: Task 1894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9fffea673e2d5f38e82e260395182e7 with result from task 1894 parsl.dataflow.dflow DEBUG: Task 1894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ba9bc927-bf4a-4dee-935a-744e8c3ea8dc_isrForCrosstalkSources_1031709_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/ba9bc927-bf4a-4dee-935a-744e8c3ea8dc_isrForCrosstalkSources_1031709_39.stderr parsl.dataflow.dflow INFO: Task 2151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33a300b9fd84f6a1794cbafc54c1b19e with result from task 2151 parsl.dataflow.dflow DEBUG: Task 2151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/432e2d7b-cb8d-4842-b9e9-55e0a96839ef_isrForCrosstalkSources_1031705_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/432e2d7b-cb8d-4842-b9e9-55e0a96839ef_isrForCrosstalkSources_1031705_5.stderr parsl.dataflow.dflow INFO: Task 2418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac71fe0aba607e2d6af8c1bb72d8fe86 with result from task 2418 parsl.dataflow.dflow DEBUG: Task 2418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/51b30b75-be33-4c34-8882-bc2617579d81_isrForCrosstalkSources_1031731_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/51b30b75-be33-4c34-8882-bc2617579d81_isrForCrosstalkSources_1031731_37.stderr parsl.dataflow.dflow INFO: Task 2368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91b6ce7f602cee2d2bdb268a749a039d with result from task 2368 parsl.dataflow.dflow DEBUG: Task 2368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/59ffdfa2-da80-4d5b-9447-d4088d5dd3b8_isrForCrosstalkSources_1031710_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/59ffdfa2-da80-4d5b-9447-d4088d5dd3b8_isrForCrosstalkSources_1031710_57.stderr 2024-03-17 04:07:08 proc_lsst.multi:170 [INFO] [multi] job_id 17148195 proc_lsst.multi INFO: [multi] job_id 17148195 2024-03-17 04:07:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 192 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 192 2024-03-17 04:07:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148195 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148195 parsl.executors.status_handling DEBUG: Launched block 247 on executor multi with job ID 17148195 parsl.executors.status_handling INFO: Allocated block ID 248 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-248 2024-03-17 04:07:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-248 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-248 2024-03-17 04:07:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 192 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 192 2024-03-17 04:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e06f8182c432425d4e79538bceaef916 with result from task 2087 parsl.dataflow.dflow DEBUG: Task 2087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/6884f24a-93a4-4694-9929-3cb5d1e0a667_isrForCrosstalkSources_1031729_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/6884f24a-93a4-4694-9929-3cb5d1e0a667_isrForCrosstalkSources_1031729_2.stderr parsl.dataflow.dflow INFO: Task 2119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9855ce9e9f103aee7ba4fc5fc615c460 with result from task 2119 parsl.dataflow.dflow DEBUG: Task 2119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/1cf7ab46-847a-4a63-82b8-2dc42d55a87b_isrForCrosstalkSources_1031717_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/1cf7ab46-847a-4a63-82b8-2dc42d55a87b_isrForCrosstalkSources_1031717_15.stderr parsl.dataflow.dflow INFO: Task 2070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0283751dd067714236a1a053c8abd36e with result from task 2070 parsl.dataflow.dflow DEBUG: Task 2070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7802e1a2-0da4-4524-8e02-3eb52f3dd189_isrForCrosstalkSources_1031710_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7802e1a2-0da4-4524-8e02-3eb52f3dd189_isrForCrosstalkSources_1031710_24.stderr parsl.dataflow.dflow INFO: Task 2324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52920f29d09b2b097eb56232437005d3 with result from task 2324 parsl.dataflow.dflow DEBUG: Task 2324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4354122d-1a4e-462c-be23-c02b1b6eacba_isrForCrosstalkSources_1031715_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/4354122d-1a4e-462c-be23-c02b1b6eacba_isrForCrosstalkSources_1031715_23.stderr parsl.dataflow.dflow INFO: Task 2118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdf19b4f511ba636270d05a3a7573f89 with result from task 2118 parsl.dataflow.dflow DEBUG: Task 2118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67c07326-bed6-46ee-952a-fd9764e1040c_isrForCrosstalkSources_1031728_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/67c07326-bed6-46ee-952a-fd9764e1040c_isrForCrosstalkSources_1031728_22.stderr parsl.dataflow.dflow INFO: Task 2114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c4de516cc822cac9b6f1d93c9767b8d with result from task 2114 parsl.dataflow.dflow DEBUG: Task 2114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/d9b02641-8c01-4dc3-922c-d02ebda04d37_isrForCrosstalkSources_1031717_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/d9b02641-8c01-4dc3-922c-d02ebda04d37_isrForCrosstalkSources_1031717_18.stderr parsl.dataflow.dflow INFO: Task 2125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d973d7a2a7f980d5d409f59989a981b9 with result from task 2125 parsl.dataflow.dflow DEBUG: Task 2125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/58aaf35f-3343-4413-8cbd-c8a7553e5afd_isrForCrosstalkSources_1031725_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/58aaf35f-3343-4413-8cbd-c8a7553e5afd_isrForCrosstalkSources_1031725_57.stderr parsl.dataflow.dflow INFO: Task 2422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7ac0892ea938de74fea97a115ba305b with result from task 2422 parsl.dataflow.dflow DEBUG: Task 2422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/d03dca95-35d3-4aaa-b80c-0be85303a9b8_isrForCrosstalkSources_1031716_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/d03dca95-35d3-4aaa-b80c-0be85303a9b8_isrForCrosstalkSources_1031716_60.stderr parsl.dataflow.dflow INFO: Task 2145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f70b18fcba228b06b8f7e8e880a05c99 with result from task 2145 parsl.dataflow.dflow DEBUG: Task 2145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ac1fb40e-00c8-4d6a-95fb-0cdcfd47b07b_isrForCrosstalkSources_1031711_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/ac1fb40e-00c8-4d6a-95fb-0cdcfd47b07b_isrForCrosstalkSources_1031711_46.stderr 2024-03-17 04:07:08 proc_lsst.multi:170 [INFO] [multi] job_id 17148196 proc_lsst.multi INFO: [multi] job_id 17148196 2024-03-17 04:07:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 193 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 193 2024-03-17 04:07:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148196 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148196 parsl.executors.status_handling DEBUG: Launched block 248 on executor multi with job ID 17148196 parsl.executors.status_handling INFO: Allocated block ID 249 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-249 2024-03-17 04:07:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-249 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-249 2024-03-17 04:07:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 193 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 193 2024-03-17 04:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f258c7d9910954b72c49fbc1d891ca13 with result from task 2158 parsl.dataflow.dflow DEBUG: Task 2158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2a08da7c-7fe3-422b-b32e-a6c599c58ad0_isrForCrosstalkSources_1031734_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/2a08da7c-7fe3-422b-b32e-a6c599c58ad0_isrForCrosstalkSources_1031734_48.stderr parsl.dataflow.dflow INFO: Task 2434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeb21b346ef819f17865098eaadccfd9 with result from task 2434 parsl.dataflow.dflow DEBUG: Task 2434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/9b0395c5-edc7-42e0-b330-7e12fd71c84b_isrForCrosstalkSources_1031700_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/9b0395c5-edc7-42e0-b330-7e12fd71c84b_isrForCrosstalkSources_1031700_13.stderr parsl.dataflow.dflow INFO: Task 1893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e168256110c54d1878f26c9ed14812f with result from task 1893 parsl.dataflow.dflow DEBUG: Task 1893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/f96534e8-dc4e-4e46-8c29-00072fdfc60a_isrForCrosstalkSources_1031702_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/f96534e8-dc4e-4e46-8c29-00072fdfc60a_isrForCrosstalkSources_1031702_36.stderr parsl.dataflow.dflow INFO: Task 1873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8cf8abedc90306dbbc6d06699997bba with result from task 1873 parsl.dataflow.dflow DEBUG: Task 1873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/24f4395b-df3b-4117-834c-256212c8ef6d_isrForCrosstalkSources_1031725_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/24f4395b-df3b-4117-834c-256212c8ef6d_isrForCrosstalkSources_1031725_2.stderr 2024-03-17 04:07:09 proc_lsst.multi:170 [INFO] [multi] job_id 17148197 proc_lsst.multi INFO: [multi] job_id 17148197 2024-03-17 04:07:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 194 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 194 2024-03-17 04:07:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148197 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148197 parsl.executors.status_handling DEBUG: Launched block 249 on executor multi with job ID 17148197 parsl.executors.status_handling INFO: Allocated block ID 250 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-250 2024-03-17 04:07:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-250 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-250 2024-03-17 04:07:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 194 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 194 2024-03-17 04:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6833dee7bed69beabb96194a32d11478 with result from task 2272 parsl.dataflow.dflow DEBUG: Task 2272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/0b442013-f2bf-43f2-bf91-14b189e8bdf0_isrForCrosstalkSources_1031699_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/0b442013-f2bf-43f2-bf91-14b189e8bdf0_isrForCrosstalkSources_1031699_14.stderr parsl.dataflow.dflow INFO: Task 2277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a430a1c5b12d81b31525f10124c84f3 with result from task 2277 parsl.dataflow.dflow DEBUG: Task 2277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c82c3b08-7fda-4e6b-82f4-8a4f6ab54a0b_isrForCrosstalkSources_1031719_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c82c3b08-7fda-4e6b-82f4-8a4f6ab54a0b_isrForCrosstalkSources_1031719_8.stderr parsl.dataflow.dflow INFO: Task 2160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92de025509e3b5223850d359b00329ab with result from task 2160 parsl.dataflow.dflow DEBUG: Task 2160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a8cc5183-715d-4e61-a090-3a6726162903_isrForCrosstalkSources_1031718_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/a8cc5183-715d-4e61-a090-3a6726162903_isrForCrosstalkSources_1031718_12.stderr parsl.dataflow.dflow INFO: Task 2165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a5196b8495ee3e5a76f8622ce625b7d with result from task 2165 parsl.dataflow.dflow DEBUG: Task 2165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/6cdab13f-4fe8-4840-a84d-cdb7b0de9a21_isrForCrosstalkSources_1031702_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/6cdab13f-4fe8-4840-a84d-cdb7b0de9a21_isrForCrosstalkSources_1031702_10.stderr parsl.dataflow.dflow INFO: Task 2352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dda15c53d29e3d346ae4f9856bc157c5 with result from task 2352 parsl.dataflow.dflow DEBUG: Task 2352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5a6a6b87-d595-4bca-8ff1-82e9f75e2007_isrForCrosstalkSources_1031705_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/5a6a6b87-d595-4bca-8ff1-82e9f75e2007_isrForCrosstalkSources_1031705_24.stderr parsl.dataflow.dflow INFO: Task 2170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c22b01ed796576c695399ebf895e636 with result from task 2170 parsl.dataflow.dflow DEBUG: Task 2170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f06a35e1-89f8-433b-b0e6-b055fb79e234_isrForCrosstalkSources_1031699_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/f06a35e1-89f8-433b-b0e6-b055fb79e234_isrForCrosstalkSources_1031699_40.stderr parsl.dataflow.dflow INFO: Task 2057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9940ed408deaaad9083fb9f70bec7491 with result from task 2057 parsl.dataflow.dflow DEBUG: Task 2057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/fa9a8503-3db9-4224-ba7d-da012d7b094f_isrForCrosstalkSources_1031705_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/fa9a8503-3db9-4224-ba7d-da012d7b094f_isrForCrosstalkSources_1031705_30.stderr parsl.dataflow.dflow INFO: Task 2380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a56ece34d3cc5e2c80796bb2f7c163b with result from task 2380 parsl.dataflow.dflow DEBUG: Task 2380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/71c53733-a64e-4b2f-a88f-f7c4362f0538_isrForCrosstalkSources_1031703_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/71c53733-a64e-4b2f-a88f-f7c4362f0538_isrForCrosstalkSources_1031703_27.stderr 2024-03-17 04:07:09 proc_lsst.multi:170 [INFO] [multi] job_id 17148198 proc_lsst.multi INFO: [multi] job_id 17148198 2024-03-17 04:07:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 195 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 195 2024-03-17 04:07:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148198 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148198 parsl.executors.status_handling DEBUG: Launched block 250 on executor multi with job ID 17148198 parsl.executors.status_handling INFO: Allocated block ID 251 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-251 2024-03-17 04:07:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-251 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-251 2024-03-17 04:07:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 195 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 195 2024-03-17 04:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4e63631a1c9f27b056ed384f4b1a766 with result from task 2270 parsl.dataflow.dflow DEBUG: Task 2270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2f7678db-6ca9-4d10-b41c-e0efa83c34f5_isrForCrosstalkSources_1031708_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/2f7678db-6ca9-4d10-b41c-e0efa83c34f5_isrForCrosstalkSources_1031708_22.stderr 2024-03-17 04:07:10 proc_lsst.multi:170 [INFO] [multi] job_id 17148199 proc_lsst.multi INFO: [multi] job_id 17148199 2024-03-17 04:07:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 196 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 196 2024-03-17 04:07:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148199 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148199 parsl.executors.status_handling DEBUG: Launched block 251 on executor multi with job ID 17148199 parsl.executors.status_handling INFO: Allocated block ID 252 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-252 2024-03-17 04:07:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-252 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-252 2024-03-17 04:07:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 196 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 196 2024-03-17 04:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 589a530426eb53747c27a1de34adf004 with result from task 2016 parsl.dataflow.dflow DEBUG: Task 2016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ccfdfa4b-c513-4567-8895-28cd961b8ec3_isrForCrosstalkSources_1031726_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/ccfdfa4b-c513-4567-8895-28cd961b8ec3_isrForCrosstalkSources_1031726_35.stderr parsl.dataflow.dflow INFO: Task 2126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f6268eeb50048aad86794af733cb846 with result from task 2126 parsl.dataflow.dflow DEBUG: Task 2126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/404bca67-edb1-425e-8914-062e0c9ba8aa_isrForCrosstalkSources_1031725_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/404bca67-edb1-425e-8914-062e0c9ba8aa_isrForCrosstalkSources_1031725_12.stderr parsl.dataflow.dflow INFO: Task 1940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc2d582d0d3145d47c281e2ea00381dd with result from task 1940 parsl.dataflow.dflow DEBUG: Task 1940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/521a28c1-3075-486b-a77e-adccaedb75aa_isrForCrosstalkSources_1031733_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/521a28c1-3075-486b-a77e-adccaedb75aa_isrForCrosstalkSources_1031733_29.stderr parsl.dataflow.dflow INFO: Task 2476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0681d679b795b1c33253e8e6fd479494 with result from task 2476 parsl.dataflow.dflow DEBUG: Task 2476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/406ced21-aeae-4e02-afda-9b86e00b8fc9_isrForCrosstalkSources_1031699_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/406ced21-aeae-4e02-afda-9b86e00b8fc9_isrForCrosstalkSources_1031699_23.stderr parsl.dataflow.dflow INFO: Task 2225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9533bde41ece82b735c9109085bbf913 with result from task 2225 parsl.dataflow.dflow DEBUG: Task 2225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ee5dff00-507f-4bab-9c7c-005ce01d1f54_isrForCrosstalkSources_1031708_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/ee5dff00-507f-4bab-9c7c-005ce01d1f54_isrForCrosstalkSources_1031708_10.stderr parsl.dataflow.dflow INFO: Task 2394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b94370fde0fbad47af639594a1ea9ef4 with result from task 2394 parsl.dataflow.dflow DEBUG: Task 2394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/db5b87d0-45a6-475d-b8f2-d922b4fb03d2_isrForCrosstalkSources_1031699_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/db5b87d0-45a6-475d-b8f2-d922b4fb03d2_isrForCrosstalkSources_1031699_22.stderr parsl.dataflow.dflow INFO: Task 2157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3724dcf152a1535de97b8dd549731d58 with result from task 2157 parsl.dataflow.dflow DEBUG: Task 2157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/bf99a6c2-3d16-4478-9984-8b0bd066d837_isrForCrosstalkSources_1031721_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/bf99a6c2-3d16-4478-9984-8b0bd066d837_isrForCrosstalkSources_1031721_38.stderr parsl.dataflow.dflow INFO: Task 2132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 360cdf5e06eff6fb5858295cb7dfd128 with result from task 2132 parsl.dataflow.dflow DEBUG: Task 2132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d3a8f67a-973d-459d-b76f-d1297b00ee9a_isrForCrosstalkSources_1031730_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/d3a8f67a-973d-459d-b76f-d1297b00ee9a_isrForCrosstalkSources_1031730_36.stderr parsl.dataflow.dflow INFO: Task 1930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db6d216eb7a23880f94b080ccb075836 with result from task 1930 parsl.dataflow.dflow DEBUG: Task 1930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e747061e-f259-4f62-b87c-38291543ddb4_isrForCrosstalkSources_1031706_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/e747061e-f259-4f62-b87c-38291543ddb4_isrForCrosstalkSources_1031706_35.stderr parsl.dataflow.dflow INFO: Task 2012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1abc9ae40232a049a0ff302825c7d8ed with result from task 2012 parsl.dataflow.dflow DEBUG: Task 2012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/36019044-aea2-444e-be17-cb824a38b7a2_isrForCrosstalkSources_1031707_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/36019044-aea2-444e-be17-cb824a38b7a2_isrForCrosstalkSources_1031707_7.stderr 2024-03-17 04:07:10 proc_lsst.multi:170 [INFO] [multi] job_id 17148200 proc_lsst.multi INFO: [multi] job_id 17148200 2024-03-17 04:07:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 197 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 197 2024-03-17 04:07:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148200 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148200 parsl.executors.status_handling DEBUG: Launched block 252 on executor multi with job ID 17148200 parsl.executors.status_handling INFO: Allocated block ID 253 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-253 2024-03-17 04:07:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-253 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-253 2024-03-17 04:07:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 197 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 197 2024-03-17 04:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a97c0f78b8db681dd4a0632e280d9ecc with result from task 2153 parsl.dataflow.dflow DEBUG: Task 2153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/f14b5276-93d7-4d8b-bb95-3bb1a0fc2f88_isrForCrosstalkSources_1031708_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/f14b5276-93d7-4d8b-bb95-3bb1a0fc2f88_isrForCrosstalkSources_1031708_11.stderr parsl.dataflow.dflow INFO: Task 2336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53588846d7ab8d4f0b1ce586732ee653 with result from task 2336 parsl.dataflow.dflow DEBUG: Task 2336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d19efe44-41c3-4348-ac03-730aff6463cb_isrForCrosstalkSources_1031734_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/d19efe44-41c3-4348-ac03-730aff6463cb_isrForCrosstalkSources_1031734_24.stderr parsl.dataflow.dflow INFO: Task 2077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f203fce52bdbb04808202f738845e62 with result from task 2077 parsl.dataflow.dflow DEBUG: Task 2077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0dc81d4d-3509-4279-b2e1-c48647d86725_isrForCrosstalkSources_1031737_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/0dc81d4d-3509-4279-b2e1-c48647d86725_isrForCrosstalkSources_1031737_3.stderr parsl.dataflow.dflow INFO: Task 2223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56ca578daf35eb45525b5596a49317af with result from task 2223 parsl.dataflow.dflow DEBUG: Task 2223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8702574b-ae98-4720-8a16-e39641a3c3ac_isrForCrosstalkSources_1031707_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/8702574b-ae98-4720-8a16-e39641a3c3ac_isrForCrosstalkSources_1031707_48.stderr parsl.dataflow.dflow INFO: Task 2094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5324edacc196167f1a1ab64a2098afd2 with result from task 2094 parsl.dataflow.dflow DEBUG: Task 2094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8f51add8-8787-4bc2-83d0-7da1c1dc1d02_isrForCrosstalkSources_1031730_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/8f51add8-8787-4bc2-83d0-7da1c1dc1d02_isrForCrosstalkSources_1031730_9.stderr parsl.dataflow.dflow INFO: Task 2259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4dfb102450f92d6ee0c4ab0332ea9d9 with result from task 2259 parsl.dataflow.dflow DEBUG: Task 2259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/a26fc7f4-7404-4d64-873d-16eb6373521c_isrForCrosstalkSources_1031705_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/a26fc7f4-7404-4d64-873d-16eb6373521c_isrForCrosstalkSources_1031705_23.stderr parsl.dataflow.dflow INFO: Task 2356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bd290e34d45544847929bda6b92aed4 with result from task 2356 parsl.dataflow.dflow DEBUG: Task 2356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/12e18dcf-b01d-4c01-93f4-d0516ef2be75_isrForCrosstalkSources_1031712_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/12e18dcf-b01d-4c01-93f4-d0516ef2be75_isrForCrosstalkSources_1031712_52.stderr 2024-03-17 04:07:11 proc_lsst.multi:170 [INFO] [multi] job_id 17148201 proc_lsst.multi INFO: [multi] job_id 17148201 2024-03-17 04:07:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 198 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 198 2024-03-17 04:07:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148201 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148201 parsl.executors.status_handling DEBUG: Launched block 253 on executor multi with job ID 17148201 parsl.executors.status_handling INFO: Allocated block ID 254 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-254 2024-03-17 04:07:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-254 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-254 2024-03-17 04:07:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 198 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 198 2024-03-17 04:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01c60726c9a161c912b555a244aa811f with result from task 2325 parsl.dataflow.dflow DEBUG: Task 2325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3bd4be64-d344-47b4-b157-864927ed407c_isrForCrosstalkSources_1031707_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3bd4be64-d344-47b4-b157-864927ed407c_isrForCrosstalkSources_1031707_18.stderr parsl.dataflow.dflow INFO: Task 2104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc218ac2bf1f1767b5473162b3a1e258 with result from task 2104 parsl.dataflow.dflow DEBUG: Task 2104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c30ba4a2-8df0-4aa5-beed-233bb1b64608_isrForCrosstalkSources_1031707_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/c30ba4a2-8df0-4aa5-beed-233bb1b64608_isrForCrosstalkSources_1031707_57.stderr parsl.dataflow.dflow INFO: Task 2128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9020c74f5f9f37ad3ee4b2a749c219b with result from task 2128 parsl.dataflow.dflow DEBUG: Task 2128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3ee931aa-0af0-4422-979c-b237ac915a8e_isrForCrosstalkSources_1031735_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/3ee931aa-0af0-4422-979c-b237ac915a8e_isrForCrosstalkSources_1031735_11.stderr parsl.dataflow.dflow INFO: Task 2280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9ab883a4163edfe23014e7972eae283 with result from task 2280 parsl.dataflow.dflow DEBUG: Task 2280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/16aa788b-6546-4014-8b03-2ff34d777d05_isrForCrosstalkSources_1031738_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/16aa788b-6546-4014-8b03-2ff34d777d05_isrForCrosstalkSources_1031738_7.stderr parsl.dataflow.dflow INFO: Task 2384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11e81f94f722ed794fdf1d20256000d4 with result from task 2384 parsl.dataflow.dflow DEBUG: Task 2384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e346a0d7-3aa5-4fa2-aa84-12c1998f418c_isrForCrosstalkSources_1031717_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/e346a0d7-3aa5-4fa2-aa84-12c1998f418c_isrForCrosstalkSources_1031717_20.stderr parsl.dataflow.dflow INFO: Task 2230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6c45a26b162bd1b4c7da8c53aa0c53c with result from task 2230 parsl.dataflow.dflow DEBUG: Task 2230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/b087c10b-225d-4c54-a0b1-342b0106ba31_isrForCrosstalkSources_1031700_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/b087c10b-225d-4c54-a0b1-342b0106ba31_isrForCrosstalkSources_1031700_10.stderr parsl.dataflow.dflow INFO: Task 2367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cfba2f16cd7bd5da193f1ab71e58a36 with result from task 2367 parsl.dataflow.dflow DEBUG: Task 2367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/023e740b-8965-431c-8c9d-ca946cb145c6_isrForCrosstalkSources_1031734_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/023e740b-8965-431c-8c9d-ca946cb145c6_isrForCrosstalkSources_1031734_42.stderr parsl.dataflow.dflow INFO: Task 1878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82640de245fe7619a27e547b46073559 with result from task 1878 parsl.dataflow.dflow DEBUG: Task 1878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/662096ea-743a-4841-a681-72d25ba37612_isrForCrosstalkSources_1031732_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/662096ea-743a-4841-a681-72d25ba37612_isrForCrosstalkSources_1031732_36.stderr parsl.dataflow.dflow INFO: Task 2154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c79e8547e3bb3d18df90f86dfa00238 with result from task 2154 parsl.dataflow.dflow DEBUG: Task 2154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c163dd45-3a39-4750-a2b4-8e4e7f80a16d_isrForCrosstalkSources_1031732_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/c163dd45-3a39-4750-a2b4-8e4e7f80a16d_isrForCrosstalkSources_1031732_9.stderr parsl.dataflow.dflow INFO: Task 2176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b597591439fea4d7b2c741e392dace15 with result from task 2176 parsl.dataflow.dflow DEBUG: Task 2176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8ac33128-0e7d-47ba-848e-c735d606de69_isrForCrosstalkSources_1031709_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8ac33128-0e7d-47ba-848e-c735d606de69_isrForCrosstalkSources_1031709_57.stderr parsl.dataflow.dflow INFO: Task 2130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddc531898e4f7a0f8d5d00538270cfa7 with result from task 2130 parsl.dataflow.dflow DEBUG: Task 2130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/37dc73a0-03b8-403e-b521-f84da81439b7_isrForCrosstalkSources_1031716_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/37dc73a0-03b8-403e-b521-f84da81439b7_isrForCrosstalkSources_1031716_5.stderr 2024-03-17 04:07:11 proc_lsst.multi:170 [INFO] [multi] job_id 17148202 proc_lsst.multi INFO: [multi] job_id 17148202 2024-03-17 04:07:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 199 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 199 2024-03-17 04:07:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148202 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148202 parsl.executors.status_handling DEBUG: Launched block 254 on executor multi with job ID 17148202 parsl.executors.status_handling INFO: Allocated block ID 255 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-255 2024-03-17 04:07:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-255 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-255 2024-03-17 04:07:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 199 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 199 2024-03-17 04:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2231e01f69302e81f506953437e2b9aa with result from task 1902 parsl.dataflow.dflow DEBUG: Task 1902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f6de5add-c54a-4f7a-a461-8bcf1a96bc77_isrForCrosstalkSources_1031721_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/f6de5add-c54a-4f7a-a461-8bcf1a96bc77_isrForCrosstalkSources_1031721_54.stderr parsl.dataflow.dflow INFO: Task 2249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c4535076425343ef1e35a4df3deafa2 with result from task 2249 parsl.dataflow.dflow DEBUG: Task 2249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a7347157-5d96-45e0-8a50-a7c962744e1e_isrForCrosstalkSources_1031731_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/a7347157-5d96-45e0-8a50-a7c962744e1e_isrForCrosstalkSources_1031731_35.stderr parsl.dataflow.dflow INFO: Task 2131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1509a1663c5ac6a15ab05da981a9361 with result from task 2131 parsl.dataflow.dflow DEBUG: Task 2131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/115e88f5-9ac5-4712-876a-bf680acae323_isrForCrosstalkSources_1031724_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/115e88f5-9ac5-4712-876a-bf680acae323_isrForCrosstalkSources_1031724_38.stderr parsl.dataflow.dflow INFO: Task 1981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec8ae11abbc9721862755d9ac78d4e09 with result from task 1981 parsl.dataflow.dflow DEBUG: Task 1981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/0a514de3-255c-4f21-9632-565287aac0be_isrForCrosstalkSources_1031718_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/0a514de3-255c-4f21-9632-565287aac0be_isrForCrosstalkSources_1031718_1.stderr parsl.dataflow.dflow INFO: Task 2292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9d9921049f25c71d87c27ad51894f2f with result from task 2292 parsl.dataflow.dflow DEBUG: Task 2292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/81225a3c-dff1-4b30-bbd2-f116db14c431_isrForCrosstalkSources_1031736_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/81225a3c-dff1-4b30-bbd2-f116db14c431_isrForCrosstalkSources_1031736_25.stderr parsl.dataflow.dflow INFO: Task 2276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7366f223aaf44149e83959fa72079cf9 with result from task 2276 parsl.dataflow.dflow DEBUG: Task 2276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8b364f63-a882-4cf8-9650-8c7fc0ab35af_isrForCrosstalkSources_1031733_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/8b364f63-a882-4cf8-9650-8c7fc0ab35af_isrForCrosstalkSources_1031733_9.stderr parsl.dataflow.dflow INFO: Task 2095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f165bd7143d6b19677a635bb46fe830 with result from task 2095 parsl.dataflow.dflow DEBUG: Task 2095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/a93a6028-9d83-46e3-9557-4440169da3f5_isrForCrosstalkSources_1031715_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/a93a6028-9d83-46e3-9557-4440169da3f5_isrForCrosstalkSources_1031715_35.stderr parsl.dataflow.dflow INFO: Task 1986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f9e5101bb8cb6031f55c66bb0cc393f with result from task 1986 parsl.dataflow.dflow DEBUG: Task 1986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/1c2d1963-0a3d-4852-8d9a-616ae19d374d_isrForCrosstalkSources_1031712_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/1c2d1963-0a3d-4852-8d9a-616ae19d374d_isrForCrosstalkSources_1031712_35.stderr parsl.dataflow.dflow INFO: Task 2011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cd1218013faa46b71267107cb820138 with result from task 2011 parsl.dataflow.dflow DEBUG: Task 2011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/bc8319a3-0133-4301-9463-64cd89a621e6_isrForCrosstalkSources_1031703_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/bc8319a3-0133-4301-9463-64cd89a621e6_isrForCrosstalkSources_1031703_55.stderr 2024-03-17 04:07:12 proc_lsst.multi:170 [INFO] [multi] job_id 17148203 proc_lsst.multi INFO: [multi] job_id 17148203 2024-03-17 04:07:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 200 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 200 2024-03-17 04:07:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148203 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148203 parsl.executors.status_handling DEBUG: Launched block 255 on executor multi with job ID 17148203 parsl.executors.status_handling INFO: Allocated block ID 256 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-256 2024-03-17 04:07:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-256 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-256 2024-03-17 04:07:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 200 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 200 2024-03-17 04:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88d36bc4a70e6d84548294135fbdf174 with result from task 1987 parsl.dataflow.dflow DEBUG: Task 1987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/19691c9c-0001-432a-acb5-86f9b162c091_isrForCrosstalkSources_1031736_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/19691c9c-0001-432a-acb5-86f9b162c091_isrForCrosstalkSources_1031736_35.stderr parsl.dataflow.dflow INFO: Task 2139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a27bfd02a6366aed533250acbb46215b with result from task 2139 parsl.dataflow.dflow DEBUG: Task 2139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/436f4b0c-ecbd-4f46-8c2e-cea3bfeb81ae_isrForCrosstalkSources_1031735_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/436f4b0c-ecbd-4f46-8c2e-cea3bfeb81ae_isrForCrosstalkSources_1031735_28.stderr parsl.dataflow.dflow INFO: Task 2478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 451de7f04cd39d9683897977bf3f6563 with result from task 2478 parsl.dataflow.dflow DEBUG: Task 2478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/27ecc693-e74f-4489-9704-4f6ca656bb9c_isrForCrosstalkSources_1031700_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/27ecc693-e74f-4489-9704-4f6ca656bb9c_isrForCrosstalkSources_1031700_12.stderr parsl.dataflow.dflow INFO: Task 2403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c0c25af806b898449dbab7ed2a35823 with result from task 2403 parsl.dataflow.dflow DEBUG: Task 2403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4c8e195b-95eb-4aa5-8d38-1c1cc67c29c3_isrForCrosstalkSources_1031706_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/4c8e195b-95eb-4aa5-8d38-1c1cc67c29c3_isrForCrosstalkSources_1031706_53.stderr parsl.dataflow.dflow INFO: Task 2058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bf47f2e1fbe60ab5066e14c7ace2e6f with result from task 2058 parsl.dataflow.dflow DEBUG: Task 2058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/d874df06-5068-4f35-ab5e-2c3ce944c0d1_isrForCrosstalkSources_1031716_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/d874df06-5068-4f35-ab5e-2c3ce944c0d1_isrForCrosstalkSources_1031716_62.stderr parsl.dataflow.dflow INFO: Task 2301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d855e37aebeecbf764813d7645554ddf with result from task 2301 parsl.dataflow.dflow DEBUG: Task 2301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/ebf3df0d-23ba-4f35-8a38-064ed7596412_isrForCrosstalkSources_1031720_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/ebf3df0d-23ba-4f35-8a38-064ed7596412_isrForCrosstalkSources_1031720_41.stderr parsl.dataflow.dflow INFO: Task 2288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee95811951f439e6f2e63751c8abdc94 with result from task 2288 parsl.dataflow.dflow DEBUG: Task 2288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/4180daed-23d1-45b9-bdb0-a773180ecfe3_isrForCrosstalkSources_1031712_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/4180daed-23d1-45b9-bdb0-a773180ecfe3_isrForCrosstalkSources_1031712_33.stderr parsl.dataflow.dflow INFO: Task 2365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb71aea115d7e2f92733df0f925631a3 with result from task 2365 parsl.dataflow.dflow DEBUG: Task 2365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/39b31961-02c8-4e73-b47e-d4870450eaf6_isrForCrosstalkSources_1031738_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/39b31961-02c8-4e73-b47e-d4870450eaf6_isrForCrosstalkSources_1031738_33.stderr parsl.dataflow.dflow INFO: Task 2174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 399852d6e48f213686cf31dbf2430285 with result from task 2174 parsl.dataflow.dflow DEBUG: Task 2174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/785fb7ef-7978-495c-b5c9-d80f561d7158_isrForCrosstalkSources_1031734_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/785fb7ef-7978-495c-b5c9-d80f561d7158_isrForCrosstalkSources_1031734_38.stderr parsl.dataflow.dflow INFO: Task 2093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfcaea23af2f410e3cc7effdf67ede5f with result from task 2093 parsl.dataflow.dflow DEBUG: Task 2093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/18e31040-61f0-46b3-be42-ed19bc548593_isrForCrosstalkSources_1031735_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/18e31040-61f0-46b3-be42-ed19bc548593_isrForCrosstalkSources_1031735_30.stderr parsl.dataflow.dflow INFO: Task 2142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fba29e75896a37ec86fd0a79b630a174 with result from task 2142 parsl.dataflow.dflow DEBUG: Task 2142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/47a22ead-5c98-4a19-a28e-c0a058c34b18_isrForCrosstalkSources_1031715_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/47a22ead-5c98-4a19-a28e-c0a058c34b18_isrForCrosstalkSources_1031715_11.stderr parsl.dataflow.dflow INFO: Task 2462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 758a35964aecc6ac2936c52aa59b13d0 with result from task 2462 parsl.dataflow.dflow DEBUG: Task 2462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/687725ec-ba68-4dc8-ac3b-dd67fdc4b163_isrForCrosstalkSources_1031700_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/687725ec-ba68-4dc8-ac3b-dd67fdc4b163_isrForCrosstalkSources_1031700_52.stderr parsl.dataflow.dflow INFO: Task 2294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eedc1ffb0e1bd1463dbeac8b240d4a08 with result from task 2294 parsl.dataflow.dflow DEBUG: Task 2294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/26c194a3-3990-4b85-a347-dd8d49ca41a5_isrForCrosstalkSources_1031726_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/26c194a3-3990-4b85-a347-dd8d49ca41a5_isrForCrosstalkSources_1031726_29.stderr parsl.dataflow.dflow INFO: Task 2369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9ed736307fdef21571143e18f178684 with result from task 2369 parsl.dataflow.dflow DEBUG: Task 2369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/38d0ef42-cd59-4d19-9944-9ee76e96b5d0_isrForCrosstalkSources_1031704_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/38d0ef42-cd59-4d19-9944-9ee76e96b5d0_isrForCrosstalkSources_1031704_27.stderr parsl.dataflow.dflow INFO: Task 2452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42dbe71ff10b2e8dca4c4ffef980430d with result from task 2452 parsl.dataflow.dflow DEBUG: Task 2452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/8416ce41-6499-4a31-a880-c9c5965c6deb_isrForCrosstalkSources_1031710_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/8416ce41-6499-4a31-a880-c9c5965c6deb_isrForCrosstalkSources_1031710_27.stderr parsl.dataflow.dflow INFO: Task 2084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 228d760c7f98f0b05d0c73ea3a2f3bf7 with result from task 2084 parsl.dataflow.dflow DEBUG: Task 2084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f775baee-7f95-4515-8146-a2fad114f238_isrForCrosstalkSources_1031706_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/f775baee-7f95-4515-8146-a2fad114f238_isrForCrosstalkSources_1031706_1.stderr parsl.dataflow.dflow INFO: Task 2252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35560b5401eee12a12e818ed0989759c with result from task 2252 parsl.dataflow.dflow DEBUG: Task 2252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/4907fc1e-e455-456b-b62a-efa9500c91e2_isrForCrosstalkSources_1031726_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/4907fc1e-e455-456b-b62a-efa9500c91e2_isrForCrosstalkSources_1031726_53.stderr 2024-03-17 04:07:12 proc_lsst.multi:170 [INFO] [multi] job_id 17148204 proc_lsst.multi INFO: [multi] job_id 17148204 2024-03-17 04:07:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 201 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 201 2024-03-17 04:07:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148204 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148204 parsl.executors.status_handling DEBUG: Launched block 256 on executor multi with job ID 17148204 parsl.executors.status_handling INFO: Allocated block ID 257 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-257 2024-03-17 04:07:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-257 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-257 2024-03-17 04:07:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 201 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 201 2024-03-17 04:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36358223100b62d919bc1bf35ecd71b4 with result from task 2385 parsl.dataflow.dflow DEBUG: Task 2385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c3294c85-cbb7-4a79-80e9-4e4036ac2ae1_isrForCrosstalkSources_1031718_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/c3294c85-cbb7-4a79-80e9-4e4036ac2ae1_isrForCrosstalkSources_1031718_30.stderr parsl.dataflow.dflow INFO: Task 2282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05dd89d22e9c830bb1277342762ff617 with result from task 2282 parsl.dataflow.dflow DEBUG: Task 2282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/ce6f2e3b-45fa-4620-819b-228fd7f20c81_isrForCrosstalkSources_1031714_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/ce6f2e3b-45fa-4620-819b-228fd7f20c81_isrForCrosstalkSources_1031714_58.stderr parsl.dataflow.dflow INFO: Task 2279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c2f642dd4e3b4f2553db6877c79fb54 with result from task 2279 parsl.dataflow.dflow DEBUG: Task 2279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/b164bfe6-9fb7-4bd0-9057-d3d3b3467efe_isrForCrosstalkSources_1031723_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/b164bfe6-9fb7-4bd0-9057-d3d3b3467efe_isrForCrosstalkSources_1031723_12.stderr parsl.dataflow.dflow INFO: Task 2227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de54e105269d18e2bfb460cc585068ac with result from task 2227 parsl.dataflow.dflow DEBUG: Task 2227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/883a0ff1-d6f6-4f94-a86e-38ba3f81a02e_isrForCrosstalkSources_1031735_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/883a0ff1-d6f6-4f94-a86e-38ba3f81a02e_isrForCrosstalkSources_1031735_52.stderr parsl.dataflow.dflow INFO: Task 2400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b076e01f08d27470e7607ab34139e6b9 with result from task 2400 parsl.dataflow.dflow DEBUG: Task 2400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/6c3ae3c1-f69f-4603-a20b-efe4d551afc6_isrForCrosstalkSources_1031734_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/6c3ae3c1-f69f-4603-a20b-efe4d551afc6_isrForCrosstalkSources_1031734_10.stderr 2024-03-17 04:07:13 proc_lsst.multi:170 [INFO] [multi] job_id 17148205 proc_lsst.multi INFO: [multi] job_id 17148205 2024-03-17 04:07:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 202 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 202 2024-03-17 04:07:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148205 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148205 parsl.executors.status_handling DEBUG: Launched block 257 on executor multi with job ID 17148205 parsl.executors.status_handling INFO: Allocated block ID 258 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-258 2024-03-17 04:07:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-258 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-258 2024-03-17 04:07:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 202 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 202 2024-03-17 04:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3a2fe1be50a69f62e5ce06fb9f6c313 with result from task 2290 parsl.dataflow.dflow DEBUG: Task 2290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/0849d87b-5e2a-4be7-a5e1-e79bca64be8a_isrForCrosstalkSources_1031708_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/0849d87b-5e2a-4be7-a5e1-e79bca64be8a_isrForCrosstalkSources_1031708_34.stderr parsl.dataflow.dflow INFO: Task 2149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69319ab0e011a9a007b07080732dd4ab with result from task 2149 parsl.dataflow.dflow DEBUG: Task 2149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/382edb67-a3a3-43ac-8b41-0b33f5fb9ea0_isrForCrosstalkSources_1031705_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/382edb67-a3a3-43ac-8b41-0b33f5fb9ea0_isrForCrosstalkSources_1031705_12.stderr parsl.dataflow.dflow INFO: Task 1879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56980c6bc0f5717e240487a10790d951 with result from task 1879 parsl.dataflow.dflow DEBUG: Task 1879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c2aeadd2-ccf2-4b79-aeac-2f16d3a8dc3e_isrForCrosstalkSources_1031731_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c2aeadd2-ccf2-4b79-aeac-2f16d3a8dc3e_isrForCrosstalkSources_1031731_58.stderr parsl.dataflow.dflow INFO: Task 1993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5aa78b4f22d7fb6df5ded98e31bf4e2 with result from task 1993 parsl.dataflow.dflow DEBUG: Task 1993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/acb19e96-f2a2-4133-b441-04039725502d_isrForCrosstalkSources_1031709_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/acb19e96-f2a2-4133-b441-04039725502d_isrForCrosstalkSources_1031709_48.stderr parsl.dataflow.dflow INFO: Task 2439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0da4c4bed5666559ef0b4604e15ec52b with result from task 2439 parsl.dataflow.dflow DEBUG: Task 2439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/78e9e8c9-d276-4028-b14c-eaaf79dd3e23_isrForCrosstalkSources_1031717_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/78e9e8c9-d276-4028-b14c-eaaf79dd3e23_isrForCrosstalkSources_1031717_14.stderr parsl.dataflow.dflow INFO: Task 2224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 992a84b1b8a099d789b4662cee917ea4 with result from task 2224 parsl.dataflow.dflow DEBUG: Task 2224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/950e4543-777e-4891-8f2e-f9615eb73835_isrForCrosstalkSources_1031735_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/950e4543-777e-4891-8f2e-f9615eb73835_isrForCrosstalkSources_1031735_29.stderr parsl.dataflow.dflow INFO: Task 1881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c2211cbbbd9e92ce5bc8e7d2731b0fd with result from task 1881 parsl.dataflow.dflow DEBUG: Task 1881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4c9ebb34-a42a-4d81-89fd-c013ab218c26_isrForCrosstalkSources_1031737_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/4c9ebb34-a42a-4d81-89fd-c013ab218c26_isrForCrosstalkSources_1031737_7.stderr parsl.dataflow.dflow INFO: Task 2424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21f4d71eba91e7ff030ea88d0be1408e with result from task 2424 parsl.dataflow.dflow DEBUG: Task 2424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3ad1a687-68e9-423c-9874-b7d04c758d51_isrForCrosstalkSources_1031719_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/3ad1a687-68e9-423c-9874-b7d04c758d51_isrForCrosstalkSources_1031719_27.stderr parsl.dataflow.dflow INFO: Task 2391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 840bffcef71a2c7acd00384237df8bd2 with result from task 2391 parsl.dataflow.dflow DEBUG: Task 2391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/b10285b6-43fc-4a33-8ae2-75d1790f6d26_isrForCrosstalkSources_1031701_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/b10285b6-43fc-4a33-8ae2-75d1790f6d26_isrForCrosstalkSources_1031701_7.stderr parsl.dataflow.dflow INFO: Task 2079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 255a39d497b4b90026d5124842414888 with result from task 2079 parsl.dataflow.dflow DEBUG: Task 2079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/f0fb0dc2-dab6-46a8-a3cf-5fdcc4b2c588_isrForCrosstalkSources_1031701_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/f0fb0dc2-dab6-46a8-a3cf-5fdcc4b2c588_isrForCrosstalkSources_1031701_30.stderr parsl.dataflow.dflow INFO: Task 2063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c30ab757c989227e707a1f360755f0a with result from task 2063 parsl.dataflow.dflow DEBUG: Task 2063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c826ecba-0bd6-45b0-90b8-18963f6f8d09_isrForCrosstalkSources_1031708_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/c826ecba-0bd6-45b0-90b8-18963f6f8d09_isrForCrosstalkSources_1031708_40.stderr parsl.dataflow.dflow INFO: Task 2147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cca593304c1ff128cef9948f05e6687 with result from task 2147 parsl.dataflow.dflow DEBUG: Task 2147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/16959245-b091-488f-a363-4f33e05bb784_isrForCrosstalkSources_1031705_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/16959245-b091-488f-a363-4f33e05bb784_isrForCrosstalkSources_1031705_59.stderr parsl.dataflow.dflow INFO: Task 1996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 238c3c0f96eca552af954f8ec07dabd6 with result from task 1996 parsl.dataflow.dflow DEBUG: Task 1996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/cfe0afc3-e682-43dd-8451-6706b9a7fae0_isrForCrosstalkSources_1031713_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/cfe0afc3-e682-43dd-8451-6706b9a7fae0_isrForCrosstalkSources_1031713_61.stderr parsl.dataflow.dflow INFO: Task 2362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12b59b8b08e7d67619aa98ed257840e2 with result from task 2362 parsl.dataflow.dflow DEBUG: Task 2362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c251b4f8-308a-4c2b-a7fb-7a62bdd8fd0f_isrForCrosstalkSources_1031712_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/c251b4f8-308a-4c2b-a7fb-7a62bdd8fd0f_isrForCrosstalkSources_1031712_19.stderr 2024-03-17 04:07:14 proc_lsst.multi:170 [INFO] [multi] job_id 17148206 proc_lsst.multi INFO: [multi] job_id 17148206 2024-03-17 04:07:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 203 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 203 2024-03-17 04:07:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148206 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148206 parsl.executors.status_handling DEBUG: Launched block 258 on executor multi with job ID 17148206 parsl.executors.status_handling INFO: Allocated block ID 259 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-259 2024-03-17 04:07:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-259 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-259 2024-03-17 04:07:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 203 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 203 2024-03-17 04:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad202af6a755d91bcfd23f4b31469cae with result from task 2121 parsl.dataflow.dflow DEBUG: Task 2121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/3e96e247-5579-4c8d-9401-5d514391972c_isrForCrosstalkSources_1031717_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/3e96e247-5579-4c8d-9401-5d514391972c_isrForCrosstalkSources_1031717_48.stderr parsl.dataflow.dflow INFO: Task 2354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86fe0fcbc24ebb19dba0f75a3ba5f2f5 with result from task 2354 parsl.dataflow.dflow DEBUG: Task 2354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fc9434b7-ab14-4d2b-a985-bf5a9b4a94e2_isrForCrosstalkSources_1031702_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/fc9434b7-ab14-4d2b-a985-bf5a9b4a94e2_isrForCrosstalkSources_1031702_39.stderr parsl.dataflow.dflow INFO: Task 2105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9fde32fdfc036e761bc87bb6d6d020a with result from task 2105 parsl.dataflow.dflow DEBUG: Task 2105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7b24ac51-30f5-4f32-978b-1cfb57d181b5_isrForCrosstalkSources_1031701_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7b24ac51-30f5-4f32-978b-1cfb57d181b5_isrForCrosstalkSources_1031701_21.stderr 2024-03-17 04:07:14 proc_lsst.multi:170 [INFO] [multi] job_id 17148207 proc_lsst.multi INFO: [multi] job_id 17148207 2024-03-17 04:07:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 204 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 204 2024-03-17 04:07:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148207 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148207 parsl.executors.status_handling DEBUG: Launched block 259 on executor multi with job ID 17148207 parsl.executors.status_handling INFO: Allocated block ID 260 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-260 2024-03-17 04:07:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-260 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-260 2024-03-17 04:07:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 204 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 204 2024-03-17 04:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 997d8078161119baa2dabb8fe045e2fd with result from task 2340 parsl.dataflow.dflow DEBUG: Task 2340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/6d1af999-7507-4d6d-be5c-de1174fcb2c3_isrForCrosstalkSources_1031731_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/6d1af999-7507-4d6d-be5c-de1174fcb2c3_isrForCrosstalkSources_1031731_21.stderr parsl.dataflow.dflow INFO: Task 2427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bf5bdbed36c864808df21a8f4f1fb5e with result from task 2427 parsl.dataflow.dflow DEBUG: Task 2427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4ca20430-aa16-4a65-99c3-871c2da8cc1e_isrForCrosstalkSources_1031717_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/4ca20430-aa16-4a65-99c3-871c2da8cc1e_isrForCrosstalkSources_1031717_40.stderr parsl.dataflow.dflow INFO: Task 2218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ee2b04e17e7dc9a1d699b057921bc5f with result from task 2218 parsl.dataflow.dflow DEBUG: Task 2218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/20adbfd3-012d-474c-84d2-685bd8d5425b_isrForCrosstalkSources_1031737_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/20adbfd3-012d-474c-84d2-685bd8d5425b_isrForCrosstalkSources_1031737_19.stderr parsl.dataflow.dflow INFO: Task 1915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 703013c656b65b6bf3e3f8fb7b8dee39 with result from task 1915 parsl.dataflow.dflow DEBUG: Task 1915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/579352d8-dc53-4626-a09c-14a7f9d0a76e_isrForCrosstalkSources_1031720_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/579352d8-dc53-4626-a09c-14a7f9d0a76e_isrForCrosstalkSources_1031720_49.stderr parsl.dataflow.dflow INFO: Task 2229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd749d1ef651033faa7cb8191649ca8c with result from task 2229 parsl.dataflow.dflow DEBUG: Task 2229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/fea3ae75-3d8e-4cfd-b452-bca1f65154cd_isrForCrosstalkSources_1031730_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/fea3ae75-3d8e-4cfd-b452-bca1f65154cd_isrForCrosstalkSources_1031730_34.stderr parsl.dataflow.dflow INFO: Task 2328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad4a7e2eb95a188e70c1ea375bab27a3 with result from task 2328 parsl.dataflow.dflow DEBUG: Task 2328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/62c38495-baf8-44dc-92b1-2def91889736_isrForCrosstalkSources_1031718_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/62c38495-baf8-44dc-92b1-2def91889736_isrForCrosstalkSources_1031718_22.stderr 2024-03-17 04:07:15 proc_lsst.multi:170 [INFO] [multi] job_id 17148208 proc_lsst.multi INFO: [multi] job_id 17148208 2024-03-17 04:07:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 205 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 205 2024-03-17 04:07:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148208 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148208 parsl.executors.status_handling DEBUG: Launched block 260 on executor multi with job ID 17148208 parsl.executors.status_handling INFO: Allocated block ID 261 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-261 2024-03-17 04:07:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-261 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-261 2024-03-17 04:07:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 205 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 205 2024-03-17 04:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77aec0096ce57baad553b948470e8ef2 with result from task 2221 parsl.dataflow.dflow DEBUG: Task 2221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/2c3d36a9-d07e-4df4-a971-60aa996d0d1a_isrForCrosstalkSources_1031722_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/2c3d36a9-d07e-4df4-a971-60aa996d0d1a_isrForCrosstalkSources_1031722_31.stderr parsl.dataflow.dflow INFO: Task 2329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 996e1fe310b4950e19b6d0c442679c73 with result from task 2329 parsl.dataflow.dflow DEBUG: Task 2329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ac0f6e5a-e758-4f74-8601-3bc6ff65026b_isrForCrosstalkSources_1031731_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/ac0f6e5a-e758-4f74-8601-3bc6ff65026b_isrForCrosstalkSources_1031731_54.stderr parsl.dataflow.dflow INFO: Task 2371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c2ec5ef72bc61214c23b5e7d16cdce4 with result from task 2371 parsl.dataflow.dflow DEBUG: Task 2371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3f53872b-cd49-4495-a067-94a60b7f59f4_isrForCrosstalkSources_1031710_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3f53872b-cd49-4495-a067-94a60b7f59f4_isrForCrosstalkSources_1031710_49.stderr 2024-03-17 04:07:15 proc_lsst.multi:170 [INFO] [multi] job_id 17148209 proc_lsst.multi INFO: [multi] job_id 17148209 2024-03-17 04:07:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 206 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 206 2024-03-17 04:07:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148209 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148209 parsl.executors.status_handling DEBUG: Launched block 261 on executor multi with job ID 17148209 parsl.executors.status_handling INFO: Allocated block ID 262 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-262 2024-03-17 04:07:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-262 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-262 2024-03-17 04:07:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 206 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 206 2024-03-17 04:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8668888dfec77d26c59defccbfbd005a with result from task 1983 parsl.dataflow.dflow DEBUG: Task 1983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c87eac10-473e-467b-96ea-6c1f42c265c8_isrForCrosstalkSources_1031716_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/c87eac10-473e-467b-96ea-6c1f42c265c8_isrForCrosstalkSources_1031716_49.stderr parsl.dataflow.dflow INFO: Task 2088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a5d929a2bada9daae26ed56bc01ec90 with result from task 2088 parsl.dataflow.dflow DEBUG: Task 2088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e33b2dc4-327e-4b9b-b542-b56c3edf1e10_isrForCrosstalkSources_1031701_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/e33b2dc4-327e-4b9b-b542-b56c3edf1e10_isrForCrosstalkSources_1031701_52.stderr parsl.dataflow.dflow INFO: Task 2159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5f4949cc62d3e9a9685538194abf344 with result from task 2159 parsl.dataflow.dflow DEBUG: Task 2159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/8f2a14af-6516-47d8-8274-05e999b43d52_isrForCrosstalkSources_1031725_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/8f2a14af-6516-47d8-8274-05e999b43d52_isrForCrosstalkSources_1031725_60.stderr parsl.dataflow.dflow INFO: Task 2323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74d34547f9c79ce62561cd250772a6b7 with result from task 2323 parsl.dataflow.dflow DEBUG: Task 2323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/203960ac-d058-4239-8e11-49b0046d2c5c_isrForCrosstalkSources_1031714_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/203960ac-d058-4239-8e11-49b0046d2c5c_isrForCrosstalkSources_1031714_54.stderr parsl.dataflow.dflow INFO: Task 2357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30d0a6c4b9e2b8350adaee6ffb514943 with result from task 2357 parsl.dataflow.dflow DEBUG: Task 2357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4a045573-bb18-436c-84a4-b2b40e966312_isrForCrosstalkSources_1031720_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4a045573-bb18-436c-84a4-b2b40e966312_isrForCrosstalkSources_1031720_60.stderr parsl.dataflow.dflow INFO: Task 2382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6943aba3f5506c72da9e4979ddafc587 with result from task 2382 parsl.dataflow.dflow DEBUG: Task 2382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/128a6f43-37fb-47c2-a8d2-394e16f100bd_isrForCrosstalkSources_1031719_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/128a6f43-37fb-47c2-a8d2-394e16f100bd_isrForCrosstalkSources_1031719_7.stderr parsl.dataflow.dflow INFO: Task 2341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbbcb3bd0a3f53df214fd7e67597acf with result from task 2341 parsl.dataflow.dflow DEBUG: Task 2341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2d1be4c6-f1f3-4215-99cf-22a0d8a62d97_isrForCrosstalkSources_1031726_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2d1be4c6-f1f3-4215-99cf-22a0d8a62d97_isrForCrosstalkSources_1031726_20.stderr 2024-03-17 04:07:16 proc_lsst.multi:170 [INFO] [multi] job_id 17148210 proc_lsst.multi INFO: [multi] job_id 17148210 2024-03-17 04:07:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 207 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 207 2024-03-17 04:07:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148210 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148210 parsl.executors.status_handling DEBUG: Launched block 262 on executor multi with job ID 17148210 parsl.executors.status_handling INFO: Allocated block ID 263 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-263 2024-03-17 04:07:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-263 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-263 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 207 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 207 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fe71c001542d60754b75b666442743f with result from task 2423 parsl.dataflow.dflow DEBUG: Task 2423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b130dde1-b0e5-48ab-9ff8-d9d41b9859b9_isrForCrosstalkSources_1031712_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b130dde1-b0e5-48ab-9ff8-d9d41b9859b9_isrForCrosstalkSources_1031712_62.stderr parsl.dataflow.dflow INFO: Task 2241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2d7b8c01c7ceaeb571eaa1a595e7ed5 with result from task 2241 parsl.dataflow.dflow DEBUG: Task 2241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5ff801d4-a4ed-4e73-8229-1a8a3ad0c96e_isrForCrosstalkSources_1031735_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/5ff801d4-a4ed-4e73-8229-1a8a3ad0c96e_isrForCrosstalkSources_1031735_26.stderr parsl.dataflow.dflow INFO: Task 2474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c169df9a39be206dc1ba0b1d14dfa90f with result from task 2474 parsl.dataflow.dflow DEBUG: Task 2474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/737cf127-4525-4b27-af16-fe28713d8691_isrForCrosstalkSources_1031722_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/737cf127-4525-4b27-af16-fe28713d8691_isrForCrosstalkSources_1031722_35.stderr parsl.dataflow.dflow INFO: Task 2297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b64bdf54906d60b00bce9ba64bc878e7 with result from task 2297 parsl.dataflow.dflow DEBUG: Task 2297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/28b29384-decf-4acd-8542-ea0f4f91c76f_isrForCrosstalkSources_1031713_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/28b29384-decf-4acd-8542-ea0f4f91c76f_isrForCrosstalkSources_1031713_39.stderr 2024-03-17 04:07:16 proc_lsst.multi:170 [INFO] [multi] job_id 17148211 proc_lsst.multi INFO: [multi] job_id 17148211 2024-03-17 04:07:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 208 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 208 2024-03-17 04:07:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148211 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148211 parsl.executors.status_handling DEBUG: Launched block 263 on executor multi with job ID 17148211 parsl.executors.status_handling INFO: Allocated block ID 264 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-264 2024-03-17 04:07:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-264 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-264 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 208 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 208 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6fc68401cd5bf295953c2a11b218107 with result from task 2266 parsl.dataflow.dflow DEBUG: Task 2266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a50a599f-7384-4290-a014-96a2f02508ce_isrForCrosstalkSources_1031711_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/a50a599f-7384-4290-a014-96a2f02508ce_isrForCrosstalkSources_1031711_40.stderr parsl.dataflow.dflow INFO: Task 2123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b79406cd25f8cb2a378ce4242b68237 with result from task 2123 parsl.dataflow.dflow DEBUG: Task 2123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/790ad14b-b5d0-4678-8f63-d0e94c7c27ca_isrForCrosstalkSources_1031733_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/790ad14b-b5d0-4678-8f63-d0e94c7c27ca_isrForCrosstalkSources_1031733_20.stderr parsl.dataflow.dflow INFO: Task 2446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 945d43fc6b8d6675180638d338c94143 with result from task 2446 parsl.dataflow.dflow DEBUG: Task 2446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/50767650-43f2-4de2-a48c-3c423542b066_isrForCrosstalkSources_1031708_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/50767650-43f2-4de2-a48c-3c423542b066_isrForCrosstalkSources_1031708_54.stderr parsl.dataflow.dflow INFO: Task 2085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e03e1eae991194145dae2bd9802b62a1 with result from task 2085 parsl.dataflow.dflow DEBUG: Task 2085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/eaf6898c-261d-4d11-a16d-18bd2024b25a_isrForCrosstalkSources_1031729_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/eaf6898c-261d-4d11-a16d-18bd2024b25a_isrForCrosstalkSources_1031729_59.stderr parsl.dataflow.dflow INFO: Task 2136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2544e8b72730b7a4406aa4725bd8c006 with result from task 2136 parsl.dataflow.dflow DEBUG: Task 2136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b9339086-0719-4080-b02e-97354c6697b2_isrForCrosstalkSources_1031712_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/b9339086-0719-4080-b02e-97354c6697b2_isrForCrosstalkSources_1031712_60.stderr parsl.dataflow.dflow INFO: Task 2163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a17071b491f7087c27f86cf37fb969c0 with result from task 2163 parsl.dataflow.dflow DEBUG: Task 2163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/40a6c531-e4e6-4fee-be7e-c0b004941cb9_isrForCrosstalkSources_1031729_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/40a6c531-e4e6-4fee-be7e-c0b004941cb9_isrForCrosstalkSources_1031729_50.stderr parsl.dataflow.dflow INFO: Task 2456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 136ab2aa5fd7f0e6c079ba797298e262 with result from task 2456 2024-03-17 04:07:16 proc_lsst.multi:170 [INFO] [multi] job_id 17148212 proc_lsst.multi INFO: [multi] job_id 17148212 2024-03-17 04:07:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 209 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 209 2024-03-17 04:07:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148212 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148212 parsl.executors.status_handling DEBUG: Launched block 264 on executor multi with job ID 17148212 parsl.executors.status_handling INFO: Allocated block ID 265 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-265 2024-03-17 04:07:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-265 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-265 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 209 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 209 2024-03-17 04:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 2456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/3e61b0d0-01e2-48fb-b69a-7fd08b72f7be_isrForCrosstalkSources_1031708_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/3e61b0d0-01e2-48fb-b69a-7fd08b72f7be_isrForCrosstalkSources_1031708_26.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c760cc60aec90cff885c52b60174c52d with result from task 2320 parsl.dataflow.dflow DEBUG: Task 2320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4df7ae11-b047-4391-a3ab-6c3100364185_isrForCrosstalkSources_1031723_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/4df7ae11-b047-4391-a3ab-6c3100364185_isrForCrosstalkSources_1031723_46.stderr parsl.dataflow.dflow INFO: Task 2366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ac36bb85174be62a16bda26c4f7a10d with result from task 2366 parsl.dataflow.dflow DEBUG: Task 2366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/90d7b8b1-3298-4c51-a0c1-35b9dbbbdbf4_isrForCrosstalkSources_1031729_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/90d7b8b1-3298-4c51-a0c1-35b9dbbbdbf4_isrForCrosstalkSources_1031729_11.stderr parsl.dataflow.dflow INFO: Task 2445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b0b97100124728b6a561ff75cdb2ee2 with result from task 2445 parsl.dataflow.dflow DEBUG: Task 2445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/570fffa8-c1ca-4a07-baef-bbaf98a94f38_isrForCrosstalkSources_1031735_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/570fffa8-c1ca-4a07-baef-bbaf98a94f38_isrForCrosstalkSources_1031735_53.stderr parsl.dataflow.dflow INFO: Task 2437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3063d2bbb69751f3cef2ef31df7a5d17 with result from task 2437 parsl.dataflow.dflow DEBUG: Task 2437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/271db4f0-e91b-4145-99a7-58e46017954a_isrForCrosstalkSources_1031717_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/271db4f0-e91b-4145-99a7-58e46017954a_isrForCrosstalkSources_1031717_26.stderr 2024-03-17 04:07:17 proc_lsst.multi:170 [INFO] [multi] job_id 17148213 proc_lsst.multi INFO: [multi] job_id 17148213 2024-03-17 04:07:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 210 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 210 2024-03-17 04:07:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148213 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148213 parsl.executors.status_handling DEBUG: Launched block 265 on executor multi with job ID 17148213 parsl.executors.status_handling INFO: Allocated block ID 266 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-266 2024-03-17 04:07:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-266 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-266 2024-03-17 04:07:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 210 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 210 2024-03-17 04:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 2289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81114027e750d5f63bd0ca8051e39e94 with result from task 2289 parsl.dataflow.dflow DEBUG: Task 2289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c1ff2c72-596c-44d0-a5b8-6cf7061ff70f_isrForCrosstalkSources_1031719_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/c1ff2c72-596c-44d0-a5b8-6cf7061ff70f_isrForCrosstalkSources_1031719_19.stderr parsl.dataflow.dflow INFO: Task 2246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e200c77444d5e3262cacca10f200c2b with result from task 2246 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 2246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/48d980db-a79e-4e1e-b055-bc17b8edec2f_isrForCrosstalkSources_1031726_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/48d980db-a79e-4e1e-b055-bc17b8edec2f_isrForCrosstalkSources_1031726_4.stderr parsl.dataflow.dflow INFO: Task 2263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8102bccb90f68a5a92585dd189d9d8e0 with result from task 2263 parsl.dataflow.dflow DEBUG: Task 2263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/83f0663f-2caf-48f4-9d6e-11ef305c15ab_isrForCrosstalkSources_1031721_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/83f0663f-2caf-48f4-9d6e-11ef305c15ab_isrForCrosstalkSources_1031721_28.stderr parsl.dataflow.dflow INFO: Task 2250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9917acf32239ab52e75e5f7451664b50 with result from task 2250 parsl.dataflow.dflow DEBUG: Task 2250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4bbb9be2-3c46-47c1-9cde-4fe09bd27f31_isrForCrosstalkSources_1031709_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/4bbb9be2-3c46-47c1-9cde-4fe09bd27f31_isrForCrosstalkSources_1031709_19.stderr parsl.dataflow.dflow INFO: Task 2296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75c212660c5907b0b46f85a1450304bd with result from task 2296 parsl.dataflow.dflow DEBUG: Task 2296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e3b7f9f8-1312-4b97-a199-5ecd6a6660ec_isrForCrosstalkSources_1031710_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/e3b7f9f8-1312-4b97-a199-5ecd6a6660ec_isrForCrosstalkSources_1031710_53.stderr parsl.dataflow.dflow INFO: Task 2374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 173f30eaa1d964fb28a72c6e1be1e0bc with result from task 2374 parsl.dataflow.dflow DEBUG: Task 2374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/70af85b9-a2da-419e-a850-4521e8a071a3_isrForCrosstalkSources_1031724_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/70af85b9-a2da-419e-a850-4521e8a071a3_isrForCrosstalkSources_1031724_48.stderr parsl.dataflow.dflow INFO: Task 2264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5463fc7b0f52a77b78a1d09cd8b7b313 with result from task 2264 parsl.dataflow.dflow DEBUG: Task 2264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2342d359-07c8-4594-9490-f6c0505e30cd_isrForCrosstalkSources_1031731_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/2342d359-07c8-4594-9490-f6c0505e30cd_isrForCrosstalkSources_1031731_5.stderr 2024-03-17 04:07:17 proc_lsst.multi:170 [INFO] [multi] job_id 17148214 proc_lsst.multi INFO: [multi] job_id 17148214 2024-03-17 04:07:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 211 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 211 2024-03-17 04:07:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148214 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148214 parsl.executors.status_handling DEBUG: Launched block 266 on executor multi with job ID 17148214 parsl.executors.status_handling INFO: Allocated block ID 267 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-267 2024-03-17 04:07:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-267 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-267 2024-03-17 04:07:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 211 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 211 2024-03-17 04:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83acf702e8793e0901c0ca1d971c6983 with result from task 2066 parsl.dataflow.dflow DEBUG: Task 2066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/c6f0adc0-3a6a-4232-b6fd-8920fa6428fd_isrForCrosstalkSources_1031733_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/c6f0adc0-3a6a-4232-b6fd-8920fa6428fd_isrForCrosstalkSources_1031733_16.stderr parsl.dataflow.dflow INFO: Task 2311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0439b2f15880d4c712237bbddb1e44cc with result from task 2311 parsl.dataflow.dflow DEBUG: Task 2311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/8f1ab18e-26a6-4303-a0c2-622d79f1a5cd_isrForCrosstalkSources_1031702_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/8f1ab18e-26a6-4303-a0c2-622d79f1a5cd_isrForCrosstalkSources_1031702_54.stderr parsl.dataflow.dflow INFO: Task 2111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b9bd052495a671c0846f88ca93b0a6c with result from task 2111 parsl.dataflow.dflow DEBUG: Task 2111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3b70ad21-16c8-4382-80b4-a4556b7a5f5a_isrForCrosstalkSources_1031736_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3b70ad21-16c8-4382-80b4-a4556b7a5f5a_isrForCrosstalkSources_1031736_8.stderr parsl.dataflow.dflow INFO: Task 2248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f14d73bf14822bf1c874e32a2f8441c0 with result from task 2248 parsl.dataflow.dflow DEBUG: Task 2248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6b2e5859-3945-46e9-9d87-01a136f865df_isrForCrosstalkSources_1031726_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/6b2e5859-3945-46e9-9d87-01a136f865df_isrForCrosstalkSources_1031726_39.stderr parsl.dataflow.dflow INFO: Task 2359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcf651b0eef22058c6d61c84a4046d79 with result from task 2359 parsl.dataflow.dflow DEBUG: Task 2359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/da824177-de8a-45e7-b763-0d7696c5af1f_isrForCrosstalkSources_1031724_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/da824177-de8a-45e7-b763-0d7696c5af1f_isrForCrosstalkSources_1031724_21.stderr parsl.providers.slurm.slurm ERROR: Submit command failed parsl.providers.slurm.slurm ERROR: Retcode:1 STDOUT: STDERR:sbatch: error: slurm_send_node_msg: [(null)] slurm_bufs_sendto(msg_type=REQUEST_SUBMIT_BATCH_JOB) failed: Unexpected missing socket error sbatch: error: Batch job submission failed: No such file or directory 2024-03-17 04:07:18 proc_lsst.multi:170 [INFO] [multi] job_id None proc_lsst.multi INFO: [multi] job_id None 2024-03-17 04:07:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 211 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 211 2024-03-17 04:07:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:18 proc_lsst.multi:175 [INFO] [multi] provider checkpoint is at capacity proc_lsst.multi INFO: [multi] provider checkpoint is at capacity parsl.executors.status_handling INFO: Allocated block ID 268 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-268 2024-03-17 04:07:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-268 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-268 2024-03-17 04:07:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 211 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 211 2024-03-17 04:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54352d681e3949e41d05c1764d0ef978 with result from task 2243 parsl.dataflow.dflow DEBUG: Task 2243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ff2592e6-ba3b-4194-88fc-9cbd70155043_isrForCrosstalkSources_1031717_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/ff2592e6-ba3b-4194-88fc-9cbd70155043_isrForCrosstalkSources_1031717_44.stderr parsl.dataflow.dflow INFO: Task 2080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1b290b46f6c124f7a1cd2ff7bdc9920 with result from task 2080 parsl.dataflow.dflow DEBUG: Task 2080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7e6f518a-011e-4a70-8ded-72bc0c6a4767_isrForCrosstalkSources_1031701_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/7e6f518a-011e-4a70-8ded-72bc0c6a4767_isrForCrosstalkSources_1031701_2.stderr parsl.dataflow.dflow INFO: Task 2315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d766ae3f150a3dea83ce1106f7a17d2 with result from task 2315 parsl.dataflow.dflow DEBUG: Task 2315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/9969de03-e9c4-4996-a09e-3801d0b9513a_isrForCrosstalkSources_1031706_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/9969de03-e9c4-4996-a09e-3801d0b9513a_isrForCrosstalkSources_1031706_12.stderr parsl.dataflow.dflow INFO: Task 2214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5179dd9ed3e2070f3311c46f74693fd4 with result from task 2214 parsl.dataflow.dflow DEBUG: Task 2214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/88497211-2e62-4619-b352-e626b561abca_isrForCrosstalkSources_1031712_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/88497211-2e62-4619-b352-e626b561abca_isrForCrosstalkSources_1031712_9.stderr parsl.dataflow.dflow INFO: Task 2284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd588ee13920713a45e2c743c841d122 with result from task 2284 parsl.dataflow.dflow DEBUG: Task 2284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3920c937-b538-441f-8a1f-873625439cb2_isrForCrosstalkSources_1031729_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/3920c937-b538-441f-8a1f-873625439cb2_isrForCrosstalkSources_1031729_46.stderr parsl.dataflow.dflow INFO: Task 2312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80fe589b0f1938dd836921d73c059f16 with result from task 2312 parsl.dataflow.dflow DEBUG: Task 2312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3aa9694f-450a-4c24-ab94-635bd2e87cda_isrForCrosstalkSources_1031707_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/3aa9694f-450a-4c24-ab94-635bd2e87cda_isrForCrosstalkSources_1031707_32.stderr parsl.dataflow.dflow INFO: Task 2431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 507f74546536b6b4c783411f37ee9fad with result from task 2431 parsl.dataflow.dflow DEBUG: Task 2431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/b7e9bc1f-02f7-43f5-9f19-6a5699929f90_isrForCrosstalkSources_1031710_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/b7e9bc1f-02f7-43f5-9f19-6a5699929f90_isrForCrosstalkSources_1031710_39.stderr parsl.dataflow.dflow INFO: Task 2333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37313bf6ed02803d03cc5f08e52feefd with result from task 2333 parsl.dataflow.dflow DEBUG: Task 2333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cfbce4b6-fed2-4a5d-ad2a-89bf6677ab1f_isrForCrosstalkSources_1031716_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031716/cfbce4b6-fed2-4a5d-ad2a-89bf6677ab1f_isrForCrosstalkSources_1031716_19.stderr parsl.dataflow.dflow INFO: Task 2407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2db64e412d3ee6ad6e8f73fbf0eed111 with result from task 2407 parsl.dataflow.dflow DEBUG: Task 2407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/d0ce25cf-b0c3-4bb2-a0a8-d0eea6d90144_isrForCrosstalkSources_1031724_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/d0ce25cf-b0c3-4bb2-a0a8-d0eea6d90144_isrForCrosstalkSources_1031724_7.stderr parsl.dataflow.dflow INFO: Task 2331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea375a0dfe476d949481007f5db6e0e9 with result from task 2331 parsl.dataflow.dflow DEBUG: Task 2331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/50ff80b2-a89a-4c4f-902f-1d8bd3075bad_isrForCrosstalkSources_1031709_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/50ff80b2-a89a-4c4f-902f-1d8bd3075bad_isrForCrosstalkSources_1031709_18.stderr parsl.dataflow.dflow INFO: Task 2064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52772f1e68890ec6fe11e645551decd2 with result from task 2064 parsl.dataflow.dflow DEBUG: Task 2064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c7cf93bf-b887-432f-9b93-c1ec2e680113_isrForCrosstalkSources_1031710_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/c7cf93bf-b887-432f-9b93-c1ec2e680113_isrForCrosstalkSources_1031710_7.stderr parsl.dataflow.dflow INFO: Task 2081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4b1214a3f3dbc5539cd36c265566c9b with result from task 2081 parsl.dataflow.dflow DEBUG: Task 2081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4f5a1ca9-6cbe-4884-b610-430ae5f3f6d6_isrForCrosstalkSources_1031720_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/4f5a1ca9-6cbe-4884-b610-430ae5f3f6d6_isrForCrosstalkSources_1031720_1.stderr parsl.dataflow.dflow INFO: Task 2373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e85861b560b9ca354e8822752687c4f4 with result from task 2373 parsl.dataflow.dflow DEBUG: Task 2373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b5beebf0-6f3f-4b5c-9984-2689d235a3a1_isrForCrosstalkSources_1031708_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b5beebf0-6f3f-4b5c-9984-2689d235a3a1_isrForCrosstalkSources_1031708_12.stderr parsl.dataflow.dflow INFO: Task 2215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80846051634fd6b9c64bbce6c8a112f0 with result from task 2215 parsl.dataflow.dflow DEBUG: Task 2215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/69d2cf43-223a-4583-8aad-4d84e6cc5086_isrForCrosstalkSources_1031724_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/69d2cf43-223a-4583-8aad-4d84e6cc5086_isrForCrosstalkSources_1031724_2.stderr parsl.dataflow.dflow INFO: Task 2360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c83d6143c52503b470896a17f0b096c with result from task 2360 parsl.dataflow.dflow DEBUG: Task 2360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/755c73ec-9226-4a7f-aa83-eb34c2752a7b_isrForCrosstalkSources_1031699_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/755c73ec-9226-4a7f-aa83-eb34c2752a7b_isrForCrosstalkSources_1031699_6.stderr parsl.dataflow.dflow INFO: Task 2364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9c38e7e8a6bcf946ec18620936b2f48 with result from task 2364 parsl.dataflow.dflow DEBUG: Task 2364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/1f9c347c-f72d-46c9-9f20-2e383895084c_isrForCrosstalkSources_1031727_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/1f9c347c-f72d-46c9-9f20-2e383895084c_isrForCrosstalkSources_1031727_7.stderr parsl.dataflow.dflow INFO: Task 2460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bb124ed8c7a5e454ab9fd693abb0ade with result from task 2460 parsl.dataflow.dflow DEBUG: Task 2460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d3d61066-d678-4e8b-857d-4f04813e2b1b_isrForCrosstalkSources_1031722_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d3d61066-d678-4e8b-857d-4f04813e2b1b_isrForCrosstalkSources_1031722_21.stderr parsl.dataflow.dflow INFO: Task 2309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0577ffced6258f5a31fec88b6f711ea5 with result from task 2309 parsl.dataflow.dflow DEBUG: Task 2309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/dd384ccd-bdaa-410c-a352-2f456911556a_isrForCrosstalkSources_1031731_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/dd384ccd-bdaa-410c-a352-2f456911556a_isrForCrosstalkSources_1031731_8.stderr parsl.dataflow.dflow INFO: Task 2363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62ba7b846d0e90e519652193b511f209 with result from task 2363 parsl.dataflow.dflow DEBUG: Task 2363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/8ff8ccde-075a-4824-9e93-05f4bb3efd56_isrForCrosstalkSources_1031727_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/8ff8ccde-075a-4824-9e93-05f4bb3efd56_isrForCrosstalkSources_1031727_54.stderr parsl.dataflow.dflow INFO: Task 2433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8821c6a122bd9d33dd9f7031d520ee71 with result from task 2433 parsl.dataflow.dflow DEBUG: Task 2433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f1c3a7b1-2c61-4b62-bc6f-b72423f68424_isrForCrosstalkSources_1031723_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/f1c3a7b1-2c61-4b62-bc6f-b72423f68424_isrForCrosstalkSources_1031723_52.stderr parsl.dataflow.dflow INFO: Task 2273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd8fd2a3a05cba159a0556cca8298a6c with result from task 2273 parsl.dataflow.dflow DEBUG: Task 2273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3f0a5c2d-1541-42bb-b310-e52e4385e7a1_isrForCrosstalkSources_1031736_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/3f0a5c2d-1541-42bb-b310-e52e4385e7a1_isrForCrosstalkSources_1031736_28.stderr parsl.dataflow.dflow INFO: Task 2256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34631ab0b47aeeaae7ccd6ddae7e0c77 with result from task 2256 parsl.dataflow.dflow DEBUG: Task 2256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/61f6d2a0-fb57-428b-8021-8f0d0ceb88e8_isrForCrosstalkSources_1031718_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/61f6d2a0-fb57-428b-8021-8f0d0ceb88e8_isrForCrosstalkSources_1031718_5.stderr parsl.dataflow.dflow INFO: Task 2255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68eff0c579b1410d1873058eed26c714 with result from task 2255 parsl.dataflow.dflow DEBUG: Task 2255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/975751e7-8e5f-45cf-b20d-67c7658ceeb8_isrForCrosstalkSources_1031712_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/975751e7-8e5f-45cf-b20d-67c7658ceeb8_isrForCrosstalkSources_1031712_3.stderr parsl.dataflow.dflow INFO: Task 2464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0fc708d3d5d09c2c90c05761d302be9 with result from task 2464 parsl.dataflow.dflow DEBUG: Task 2464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3011c9a9-3b26-4394-820c-8e89e45b2318_isrForCrosstalkSources_1031710_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/3011c9a9-3b26-4394-820c-8e89e45b2318_isrForCrosstalkSources_1031710_11.stderr parsl.dataflow.dflow INFO: Task 2281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1bad6e70e0c016cdeb7242da25c5cc5 with result from task 2281 parsl.dataflow.dflow DEBUG: Task 2281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/fffc5bae-a33b-44a5-ac6c-c0fc23f7b84a_isrForCrosstalkSources_1031703_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/fffc5bae-a33b-44a5-ac6c-c0fc23f7b84a_isrForCrosstalkSources_1031703_30.stderr 2024-03-17 04:07:20 proc_lsst.multi:170 [INFO] [multi] job_id 17148215 proc_lsst.multi INFO: [multi] job_id 17148215 2024-03-17 04:07:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 212 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 212 2024-03-17 04:07:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148215 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148215 parsl.executors.status_handling DEBUG: Launched block 268 on executor multi with job ID 17148215 parsl.executors.status_handling INFO: Allocated block ID 269 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-269 2024-03-17 04:07:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-269 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-269 2024-03-17 04:07:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 212 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 212 2024-03-17 04:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 2461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd8d49a21720ce112c135e0a79be74ab with result from task 2461 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 2461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/13d324b4-bfc5-44c9-a632-cf1d5d025963_isrForCrosstalkSources_1031715_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/13d324b4-bfc5-44c9-a632-cf1d5d025963_isrForCrosstalkSources_1031715_18.stderr parsl.dataflow.dflow INFO: Task 2388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d3062349494b1e7b550744cc9ec8106 with result from task 2388 parsl.dataflow.dflow DEBUG: Task 2388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c295276d-789c-4916-ad27-f15dfb2bf5fd_isrForCrosstalkSources_1031728_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/c295276d-789c-4916-ad27-f15dfb2bf5fd_isrForCrosstalkSources_1031728_4.stderr parsl.dataflow.dflow INFO: Task 2261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9109ff728abc51aa653ff7ab8187e6ec with result from task 2261 parsl.dataflow.dflow DEBUG: Task 2261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/729dd954-d477-42a7-98fd-f50cbf2b8812_isrForCrosstalkSources_1031727_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/729dd954-d477-42a7-98fd-f50cbf2b8812_isrForCrosstalkSources_1031727_21.stderr parsl.dataflow.dflow INFO: Task 2406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09ef5eec4a9968225e59ff0293d06cf2 with result from task 2406 parsl.dataflow.dflow DEBUG: Task 2406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c8239655-0eb0-45ef-bd0a-51a4c9f00f5a_isrForCrosstalkSources_1031734_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c8239655-0eb0-45ef-bd0a-51a4c9f00f5a_isrForCrosstalkSources_1031734_9.stderr parsl.dataflow.dflow INFO: Task 2408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a0e5da993a97f66f7900b5cd8cede2a with result from task 2408 parsl.dataflow.dflow DEBUG: Task 2408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/70709163-cb79-4c21-b161-5c61ba7d1de4_isrForCrosstalkSources_1031722_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/70709163-cb79-4c21-b161-5c61ba7d1de4_isrForCrosstalkSources_1031722_50.stderr parsl.dataflow.dflow INFO: Task 2271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73e8170fc45a506aa41da418fa28a5b7 with result from task 2271 parsl.dataflow.dflow DEBUG: Task 2271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/86b7ba8a-232c-480d-bccf-61093d2d7cf8_isrForCrosstalkSources_1031703_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/86b7ba8a-232c-480d-bccf-61093d2d7cf8_isrForCrosstalkSources_1031703_43.stderr parsl.dataflow.dflow INFO: Task 2269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a703e5f172db4a06a296fe179da14be with result from task 2269 parsl.dataflow.dflow DEBUG: Task 2269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/09f57c58-32e2-4aab-99d2-d68aca0a8826_isrForCrosstalkSources_1031735_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/09f57c58-32e2-4aab-99d2-d68aca0a8826_isrForCrosstalkSources_1031735_2.stderr parsl.dataflow.dflow INFO: Task 2479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95d6d914895ffecdfa51977cc62e2a87 with result from task 2479 parsl.dataflow.dflow DEBUG: Task 2479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/db6a62ef-597e-4eed-b96f-6c727b3a085a_isrForCrosstalkSources_1031701_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/db6a62ef-597e-4eed-b96f-6c727b3a085a_isrForCrosstalkSources_1031701_10.stderr parsl.dataflow.dflow INFO: Task 2417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9973b5955d415a11b784e831368f93bf with result from task 2417 parsl.dataflow.dflow DEBUG: Task 2417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/0774314b-8463-470d-a7ad-83657c8fc6c9_isrForCrosstalkSources_1031710_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/0774314b-8463-470d-a7ad-83657c8fc6c9_isrForCrosstalkSources_1031710_15.stderr parsl.dataflow.dflow INFO: Task 2444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a1fafab115daf717e5752aaf7a1271c with result from task 2444 parsl.dataflow.dflow DEBUG: Task 2444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/93956d27-79d4-475e-81b9-210ee3e66474_isrForCrosstalkSources_1031729_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/93956d27-79d4-475e-81b9-210ee3e66474_isrForCrosstalkSources_1031729_12.stderr parsl.dataflow.dflow INFO: Task 2443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcae39da7e5404844e4fa78e003aaeef with result from task 2443 parsl.dataflow.dflow DEBUG: Task 2443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2e86bc71-9778-497d-8b5b-59184ed05393_isrForCrosstalkSources_1031726_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/2e86bc71-9778-497d-8b5b-59184ed05393_isrForCrosstalkSources_1031726_51.stderr parsl.dataflow.dflow INFO: Task 2395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec80f6bd69080810d9939b693f1ab6b5 with result from task 2395 parsl.dataflow.dflow DEBUG: Task 2395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/0119114b-7ae1-414a-bb99-0c1f6ddfd4a0_isrForCrosstalkSources_1031735_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/0119114b-7ae1-414a-bb99-0c1f6ddfd4a0_isrForCrosstalkSources_1031735_18.stderr parsl.dataflow.dflow INFO: Task 2451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e10c991f0919e4e43c8812c1b06321b7 with result from task 2451 parsl.dataflow.dflow DEBUG: Task 2451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2d6e5eaf-d713-45b6-97a5-142a24a131d8_isrForCrosstalkSources_1031709_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/2d6e5eaf-d713-45b6-97a5-142a24a131d8_isrForCrosstalkSources_1031709_51.stderr parsl.dataflow.dflow INFO: Task 2260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5ce8cda040bc241f05816b16458b20e with result from task 2260 parsl.dataflow.dflow DEBUG: Task 2260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/996b35c8-e264-4486-ac26-03e177d3f90f_isrForCrosstalkSources_1031702_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/996b35c8-e264-4486-ac26-03e177d3f90f_isrForCrosstalkSources_1031702_30.stderr parsl.dataflow.dflow INFO: Task 2278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e22fa07c1692a76a6f445682dd5af29e with result from task 2278 parsl.dataflow.dflow DEBUG: Task 2278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/390aa2e0-2c71-46c2-ace2-0f72a90fc77a_isrForCrosstalkSources_1031705_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/390aa2e0-2c71-46c2-ace2-0f72a90fc77a_isrForCrosstalkSources_1031705_48.stderr parsl.dataflow.dflow INFO: Task 2275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce3f0547c9bdfe1f1853626533bf8218 with result from task 2275 parsl.dataflow.dflow DEBUG: Task 2275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/45b01147-3914-4346-9aa1-f450ff5f434e_isrForCrosstalkSources_1031702_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/45b01147-3914-4346-9aa1-f450ff5f434e_isrForCrosstalkSources_1031702_55.stderr parsl.dataflow.dflow INFO: Task 2283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3c3783551476dd624061bbe50fe4d48 with result from task 2283 parsl.dataflow.dflow DEBUG: Task 2283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/0bd20edf-7a4f-44d8-9f7b-72767cdef6f2_isrForCrosstalkSources_1031731_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/0bd20edf-7a4f-44d8-9f7b-72767cdef6f2_isrForCrosstalkSources_1031731_6.stderr parsl.dataflow.dflow INFO: Task 2420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3b8090e5da9bfef74549e2dcc4e7c77 with result from task 2420 parsl.dataflow.dflow DEBUG: Task 2420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/69e9c047-be15-4848-b139-83e160817cde_isrForCrosstalkSources_1031736_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/69e9c047-be15-4848-b139-83e160817cde_isrForCrosstalkSources_1031736_47.stderr parsl.dataflow.dflow INFO: Task 2425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f20f3c26f37485e68293c4d275384672 with result from task 2425 parsl.dataflow.dflow DEBUG: Task 2425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/cbabcf33-d27e-4707-9cb1-a34a2b3926da_isrForCrosstalkSources_1031719_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/cbabcf33-d27e-4707-9cb1-a34a2b3926da_isrForCrosstalkSources_1031719_10.stderr parsl.dataflow.dflow INFO: Task 2469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28b6ae015b063b9feb42f585a545d067 with result from task 2469 parsl.dataflow.dflow DEBUG: Task 2469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/a67c1242-be4b-4880-85a3-49aa85a3dbaf_isrForCrosstalkSources_1031737_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/a67c1242-be4b-4880-85a3-49aa85a3dbaf_isrForCrosstalkSources_1031737_4.stderr parsl.dataflow.dflow INFO: Task 2432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cc451908175337f621872ca9f0ca2ad with result from task 2432 parsl.dataflow.dflow DEBUG: Task 2432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/1e8cbec1-78da-4f4d-8350-2f1a08484502_isrForCrosstalkSources_1031712_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/1e8cbec1-78da-4f4d-8350-2f1a08484502_isrForCrosstalkSources_1031712_12.stderr parsl.dataflow.dflow INFO: Task 2372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ee9e50f0ae888c9d6f0fbd5990465c8 with result from task 2372 parsl.dataflow.dflow DEBUG: Task 2372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/eac392ab-b49d-472d-9749-2e47979a850b_isrForCrosstalkSources_1031717_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/eac392ab-b49d-472d-9749-2e47979a850b_isrForCrosstalkSources_1031717_53.stderr parsl.dataflow.dflow INFO: Task 2468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e42196e6d089af6acef56267fc43919 with result from task 2468 parsl.dataflow.dflow DEBUG: Task 2468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/98915ee4-2029-4271-950f-5493ed2f6c28_isrForCrosstalkSources_1031708_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/98915ee4-2029-4271-950f-5493ed2f6c28_isrForCrosstalkSources_1031708_7.stderr parsl.dataflow.dflow INFO: Task 2351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a624d34c2943b7c8cb45901491bb1556 with result from task 2351 parsl.dataflow.dflow DEBUG: Task 2351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/7819bb82-2a37-4728-9b10-0c2b3885650f_isrForCrosstalkSources_1031720_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/7819bb82-2a37-4728-9b10-0c2b3885650f_isrForCrosstalkSources_1031720_45.stderr parsl.dataflow.dflow INFO: Task 2457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 063d9807a1ef039b6fe2600111d2d50c with result from task 2457 parsl.dataflow.dflow DEBUG: Task 2457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0fcd7c13-d27f-41aa-af8a-2b3f0be1f254_isrForCrosstalkSources_1031707_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0fcd7c13-d27f-41aa-af8a-2b3f0be1f254_isrForCrosstalkSources_1031707_36.stderr parsl.dataflow.dflow INFO: Task 2262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25267870a90cd6262044ad535b667879 with result from task 2262 parsl.dataflow.dflow DEBUG: Task 2262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/308f9690-e667-43a6-855d-cc3ca393802d_isrForCrosstalkSources_1031708_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/308f9690-e667-43a6-855d-cc3ca393802d_isrForCrosstalkSources_1031708_47.stderr parsl.dataflow.dflow INFO: Task 2401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9284dd74b3c0795c34cd16a5e0c26dbd with result from task 2401 parsl.dataflow.dflow DEBUG: Task 2401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/22cb06e0-9500-4299-b35e-6512bd46ea29_isrForCrosstalkSources_1031722_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/22cb06e0-9500-4299-b35e-6512bd46ea29_isrForCrosstalkSources_1031722_49.stderr parsl.dataflow.dflow INFO: Task 2332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8e74acb124d9308edbc78dfd4fe9303 with result from task 2332 parsl.dataflow.dflow DEBUG: Task 2332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/03b37c98-ab10-4d02-bda9-7839f8771626_isrForCrosstalkSources_1031707_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/03b37c98-ab10-4d02-bda9-7839f8771626_isrForCrosstalkSources_1031707_51.stderr parsl.dataflow.dflow INFO: Task 2216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc1a0242daa5ff96197dcaff5b1545fb with result from task 2216 parsl.dataflow.dflow DEBUG: Task 2216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8c426a0c-9495-4d4c-bdfa-8de5a822b236_isrForCrosstalkSources_1031703_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/8c426a0c-9495-4d4c-bdfa-8de5a822b236_isrForCrosstalkSources_1031703_45.stderr parsl.dataflow.dflow INFO: Task 2298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26f405ba4ad7d5314bdb4e5e2ddd4c34 with result from task 2298 parsl.dataflow.dflow DEBUG: Task 2298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/c52f9ccd-13a2-4dcc-b50c-2c6077be4f54_isrForCrosstalkSources_1031699_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/c52f9ccd-13a2-4dcc-b50c-2c6077be4f54_isrForCrosstalkSources_1031699_41.stderr parsl.dataflow.dflow INFO: Task 2416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 920c996876e26d980cd0a3953c3376e0 with result from task 2416 parsl.dataflow.dflow DEBUG: Task 2416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/20f02faa-2f46-408b-9e20-82689bca5ad0_isrForCrosstalkSources_1031726_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/20f02faa-2f46-408b-9e20-82689bca5ad0_isrForCrosstalkSources_1031726_11.stderr parsl.dataflow.dflow INFO: Task 2453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e87ad20df55d8758361771367045143c with result from task 2453 parsl.dataflow.dflow DEBUG: Task 2453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/16518ba9-1593-4029-8c66-cb6f5efd3e4e_isrForCrosstalkSources_1031699_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/16518ba9-1593-4029-8c66-cb6f5efd3e4e_isrForCrosstalkSources_1031699_55.stderr parsl.dataflow.dflow INFO: Task 2379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c426d9dcf4b5e00d0696fdb278bb1f1 with result from task 2379 parsl.dataflow.dflow DEBUG: Task 2379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/b22631c9-df58-47da-9b4d-bd6fa3ef7fcc_isrForCrosstalkSources_1031699_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/b22631c9-df58-47da-9b4d-bd6fa3ef7fcc_isrForCrosstalkSources_1031699_49.stderr parsl.dataflow.dflow INFO: Task 1959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 702c9241f3bffb957a52453378c5a68b with result from task 1959 parsl.dataflow.dflow DEBUG: Task 1959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0327063a-fa4d-4033-8165-b729e9e777c6_isrForCrosstalkSources_1031705_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/0327063a-fa4d-4033-8165-b729e9e777c6_isrForCrosstalkSources_1031705_44.stderr parsl.dataflow.dflow INFO: Task 1962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 433645a99d6ee2487d414b404d4aac6e with result from task 1962 parsl.dataflow.dflow DEBUG: Task 1962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/898b2c30-8e40-47ae-aa31-1f2c45753bcf_isrForCrosstalkSources_1031730_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/898b2c30-8e40-47ae-aa31-1f2c45753bcf_isrForCrosstalkSources_1031730_55.stderr parsl.dataflow.dflow INFO: Task 1956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0aad658006a8994ccf751b6a43d0050f with result from task 1956 parsl.dataflow.dflow DEBUG: Task 1956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/31b4e2f2-2529-41d6-b43f-6621826a0a9a_isrForCrosstalkSources_1031718_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/31b4e2f2-2529-41d6-b43f-6621826a0a9a_isrForCrosstalkSources_1031718_47.stderr parsl.dataflow.dflow INFO: Task 1968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3728d9d29487e7a555ce59d40d6d490 with result from task 1968 parsl.dataflow.dflow DEBUG: Task 1968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c598bdf5-69f1-49e9-8000-1c87d7d1ca08_isrForCrosstalkSources_1031734_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/c598bdf5-69f1-49e9-8000-1c87d7d1ca08_isrForCrosstalkSources_1031734_7.stderr parsl.dataflow.dflow INFO: Task 2265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9aeccc80986b751e7e6b8c29c615c28b with result from task 2265 parsl.dataflow.dflow DEBUG: Task 2265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6942f277-03e7-4e77-a0ed-a14bac9fc9a3_isrForCrosstalkSources_1031737_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/6942f277-03e7-4e77-a0ed-a14bac9fc9a3_isrForCrosstalkSources_1031737_41.stderr parsl.dataflow.dflow INFO: Task 1976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92c65936f006eecaa8140f13030328fb with result from task 1976 parsl.dataflow.dflow DEBUG: Task 1976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/257064e0-8eb0-41e4-b36d-d18eef0caa84_isrForCrosstalkSources_1031736_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/257064e0-8eb0-41e4-b36d-d18eef0caa84_isrForCrosstalkSources_1031736_58.stderr parsl.dataflow.dflow INFO: Task 1975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89478b60b642466f60e0a9189e93c819 with result from task 1975 parsl.dataflow.dflow DEBUG: Task 1975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7d325956-bea5-4a0a-b38f-066f82c8e93b_isrForCrosstalkSources_1031732_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/7d325956-bea5-4a0a-b38f-066f82c8e93b_isrForCrosstalkSources_1031732_50.stderr parsl.dataflow.dflow INFO: Task 1964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a530f8a00cc3961947020e89be330119 with result from task 1964 parsl.dataflow.dflow DEBUG: Task 1964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2a0a9efc-d769-4519-84ef-ae6d3dc7b09b_isrForCrosstalkSources_1031723_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/2a0a9efc-d769-4519-84ef-ae6d3dc7b09b_isrForCrosstalkSources_1031723_32.stderr parsl.dataflow.dflow INFO: Task 1972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f88719c6fb76bbbbfe52b9c9f55429e with result from task 1972 parsl.dataflow.dflow DEBUG: Task 1972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d564906b-57de-49d3-8636-638bd757cac6_isrForCrosstalkSources_1031711_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/d564906b-57de-49d3-8636-638bd757cac6_isrForCrosstalkSources_1031711_56.stderr parsl.dataflow.dflow INFO: Task 1977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc11ec9d9adb8e3ec2ec328872d81c49 with result from task 1977 parsl.dataflow.dflow DEBUG: Task 1977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/32307ce9-f935-4382-9c9f-869446ae1d92_isrForCrosstalkSources_1031736_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/32307ce9-f935-4382-9c9f-869446ae1d92_isrForCrosstalkSources_1031736_48.stderr parsl.dataflow.dflow INFO: Task 1974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7217245593fafe75247a6456cdddf982 with result from task 1974 parsl.dataflow.dflow DEBUG: Task 1974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/937037fb-2cc1-42f6-8761-8d9dffbc61d3_isrForCrosstalkSources_1031711_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/937037fb-2cc1-42f6-8761-8d9dffbc61d3_isrForCrosstalkSources_1031711_6.stderr parsl.dataflow.dflow INFO: Task 1957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfc3a0740ef371c82ab77ede0ac29909 with result from task 1957 parsl.dataflow.dflow DEBUG: Task 1957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/afca63e1-fa5c-4e0d-a2d1-1bf6253f68c3_isrForCrosstalkSources_1031704_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/afca63e1-fa5c-4e0d-a2d1-1bf6253f68c3_isrForCrosstalkSources_1031704_15.stderr parsl.dataflow.dflow INFO: Task 1960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 317bae838117e9d5a42f011b253701d7 with result from task 1960 parsl.dataflow.dflow DEBUG: Task 1960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6814d677-9239-4c5b-8368-612c71f6566c_isrForCrosstalkSources_1031711_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/6814d677-9239-4c5b-8368-612c71f6566c_isrForCrosstalkSources_1031711_24.stderr parsl.dataflow.dflow INFO: Task 1958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 312d031a8fb11c989d8ee1daaae57f7e with result from task 1958 parsl.dataflow.dflow DEBUG: Task 1958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/8d7621c8-adc4-404c-a1e6-e62ede0b36d8_isrForCrosstalkSources_1031731_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/8d7621c8-adc4-404c-a1e6-e62ede0b36d8_isrForCrosstalkSources_1031731_45.stderr parsl.dataflow.dflow INFO: Task 1966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36fa9d8063e82edc2fb476578682146e with result from task 1966 parsl.dataflow.dflow DEBUG: Task 1966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/837b41e9-5d6d-4982-82d9-1226e38a8d4c_isrForCrosstalkSources_1031704_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/837b41e9-5d6d-4982-82d9-1226e38a8d4c_isrForCrosstalkSources_1031704_25.stderr parsl.dataflow.dflow INFO: Task 1970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34ba998923222c39ddecdeb3d1038bfe with result from task 1970 parsl.dataflow.dflow DEBUG: Task 1970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/77819fba-e673-43a1-9a38-f21c9bd45c66_isrForCrosstalkSources_1031734_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/77819fba-e673-43a1-9a38-f21c9bd45c66_isrForCrosstalkSources_1031734_30.stderr parsl.dataflow.dflow INFO: Task 1971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f996320fc61138f769bfcb1e611ece3 with result from task 1971 parsl.dataflow.dflow DEBUG: Task 1971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks 2024-03-17 04:07:23 proc_lsst.multi:170 [INFO] [multi] job_id 17148216 proc_lsst.multi INFO: [multi] job_id 17148216 2024-03-17 04:07:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 213 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 213 2024-03-17 04:07:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148216 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148216 parsl.dataflow.dflow INFO: Standard output for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8987509d-b10b-47ca-a2c5-d92bb7dd9b1d_isrForCrosstalkSources_1031699_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/8987509d-b10b-47ca-a2c5-d92bb7dd9b1d_isrForCrosstalkSources_1031699_50.stderr parsl.dataflow.dflow INFO: Task 1969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37174c5efa547a0c7c3bee58c96af44f with result from task 1969 parsl.executors.status_handling DEBUG: Launched block 269 on executor multi with job ID 17148216 parsl.executors.status_handling INFO: Allocated block ID 270 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-270 2024-03-17 04:07:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-270 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-270 2024-03-17 04:07:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 213 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 213 2024-03-17 04:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 1969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/8899245d-46c3-42fc-8e90-bce0ddf9a188_isrForCrosstalkSources_1031715_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/8899245d-46c3-42fc-8e90-bce0ddf9a188_isrForCrosstalkSources_1031715_33.stderr parsl.dataflow.dflow INFO: Task 1961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16ecf20f57b8861a7a61ec788251bbf0 with result from task 1961 parsl.dataflow.dflow DEBUG: Task 1961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/97114f2d-f8ab-4c6e-8315-7c0c89d495cb_isrForCrosstalkSources_1031734_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031734/97114f2d-f8ab-4c6e-8315-7c0c89d495cb_isrForCrosstalkSources_1031734_28.stderr parsl.dataflow.dflow INFO: Task 1973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f15d33265415bfe9b9431b4fe2892d8e with result from task 1973 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 1973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/70e740cb-16cf-4b9a-94b0-f33cd5525cb5_isrForCrosstalkSources_1031737_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031737/70e740cb-16cf-4b9a-94b0-f33cd5525cb5_isrForCrosstalkSources_1031737_28.stderr parsl.dataflow.dflow INFO: Task 1979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9fad07e759a0c93fbfe7b9801999e18 with result from task 1979 parsl.dataflow.dflow DEBUG: Task 1979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e434a537-7ab7-41bd-9fa6-a0f73180aba3_isrForCrosstalkSources_1031727_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/e434a537-7ab7-41bd-9fa6-a0f73180aba3_isrForCrosstalkSources_1031727_22.stderr parsl.dataflow.dflow INFO: Task 2245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a35ac5e5ce06397586c223a9bf1f628 with result from task 2245 parsl.dataflow.dflow DEBUG: Task 2245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/4929f560-e715-48eb-8df7-ee0f317d15c8_isrForCrosstalkSources_1031702_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/4929f560-e715-48eb-8df7-ee0f317d15c8_isrForCrosstalkSources_1031702_56.stderr parsl.dataflow.dflow INFO: Task 1965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 647177509c5d3f20b704f0c5ebfe843a with result from task 1965 parsl.dataflow.dflow DEBUG: Task 1965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/4685e0a9-c44c-4032-87e6-24e9732b37d5_isrForCrosstalkSources_1031730_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/4685e0a9-c44c-4032-87e6-24e9732b37d5_isrForCrosstalkSources_1031730_20.stderr parsl.dataflow.dflow INFO: Task 1963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5f81a8ad700b3d6373bf37514af8308 with result from task 1963 parsl.dataflow.dflow DEBUG: Task 1963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/db918598-354c-48a7-ad69-29876b9793f4_isrForCrosstalkSources_1031724_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/db918598-354c-48a7-ad69-29876b9793f4_isrForCrosstalkSources_1031724_11.stderr parsl.dataflow.dflow INFO: Task 2450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd18a500120b18cf3567bc18b553b0f3 with result from task 2450 parsl.dataflow.dflow DEBUG: Task 2450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/36a2f6f2-379b-4fb1-a80b-f7e2eb7ce1fd_isrForCrosstalkSources_1031699_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031699/36a2f6f2-379b-4fb1-a80b-f7e2eb7ce1fd_isrForCrosstalkSources_1031699_47.stderr parsl.dataflow.dflow INFO: Task 2134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f71fb6fd611e3ca89bdf1104f56f4faa with result from task 2134 parsl.dataflow.dflow DEBUG: Task 2134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/eec07911-cf50-4c60-920c-a5a3f6717894_isrForCrosstalkSources_1031715_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/eec07911-cf50-4c60-920c-a5a3f6717894_isrForCrosstalkSources_1031715_59.stderr parsl.dataflow.dflow INFO: Task 2069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62f9713da48a2664fec4121ce4505991 with result from task 2069 parsl.dataflow.dflow DEBUG: Task 2069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/a728f267-1b74-431b-8d62-8af9c469a1c1_isrForCrosstalkSources_1031703_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/a728f267-1b74-431b-8d62-8af9c469a1c1_isrForCrosstalkSources_1031703_57.stderr parsl.dataflow.dflow INFO: Task 2335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a91b0e8c7adb0775a8cad696cba51528 with result from task 2335 parsl.dataflow.dflow DEBUG: Task 2335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/22e43c29-ac28-476e-acc6-b28ca2d062c5_isrForCrosstalkSources_1031704_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/22e43c29-ac28-476e-acc6-b28ca2d062c5_isrForCrosstalkSources_1031704_18.stderr parsl.dataflow.dflow INFO: Task 1967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eecb2fbcde0b86f84f6cb1a6fff6bc98 with result from task 1967 parsl.dataflow.dflow DEBUG: Task 1967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/edb6cc17-5b03-4e3a-a9d7-09c657e9a772_isrForCrosstalkSources_1031713_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/edb6cc17-5b03-4e3a-a9d7-09c657e9a772_isrForCrosstalkSources_1031713_36.stderr parsl.dataflow.dflow INFO: Task 2383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c069ea210291cfc23858becb1f9c99c7 with result from task 2383 parsl.dataflow.dflow DEBUG: Task 2383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/1039694b-25da-4b98-a066-c26552fa420e_isrForCrosstalkSources_1031723_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/1039694b-25da-4b98-a066-c26552fa420e_isrForCrosstalkSources_1031723_15.stderr parsl.dataflow.dflow INFO: Task 2353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c488847cc601d9e7c5f9d5f0ee5c902e with result from task 2353 parsl.dataflow.dflow DEBUG: Task 2353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e7bf91ec-e0d5-48aa-9e6f-589b27c0d8e8_isrForCrosstalkSources_1031725_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/e7bf91ec-e0d5-48aa-9e6f-589b27c0d8e8_isrForCrosstalkSources_1031725_1.stderr parsl.dataflow.dflow INFO: Task 2295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5885ee2ca971febac9474b5bd268332 with result from task 2295 parsl.dataflow.dflow DEBUG: Task 2295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/d16902bb-1586-4fa4-b92a-2c387d862085_isrForCrosstalkSources_1031708_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/d16902bb-1586-4fa4-b92a-2c387d862085_isrForCrosstalkSources_1031708_21.stderr parsl.dataflow.dflow INFO: Task 2467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8e302192a98720756fcfe4b2e065c44 with result from task 2467 parsl.dataflow.dflow DEBUG: Task 2467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/3d8e5ee7-28cf-4481-879e-792b70323bb9_isrForCrosstalkSources_1031720_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/3d8e5ee7-28cf-4481-879e-792b70323bb9_isrForCrosstalkSources_1031720_3.stderr parsl.dataflow.dflow INFO: Task 2386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96535e9461fd655fc7df13f4b57a784c with result from task 2386 parsl.dataflow.dflow DEBUG: Task 2386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/94a656dc-cc60-4b17-85b5-795369ac05fc_isrForCrosstalkSources_1031720_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/94a656dc-cc60-4b17-85b5-795369ac05fc_isrForCrosstalkSources_1031720_22.stderr parsl.dataflow.dflow INFO: Task 2347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66b723a4535c638d1d3155b52078411a with result from task 2347 parsl.dataflow.dflow DEBUG: Task 2347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/19159d90-4f13-47d4-8c87-5af963cddfb4_isrForCrosstalkSources_1031704_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/19159d90-4f13-47d4-8c87-5af963cddfb4_isrForCrosstalkSources_1031704_52.stderr parsl.dataflow.dflow INFO: Task 2293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42a7f9fff64e28c56355d80e3c7796c7 with result from task 2293 parsl.dataflow.dflow DEBUG: Task 2293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b5f34b82-4df3-44f5-a3f3-c8418b1ab601_isrForCrosstalkSources_1031713_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/b5f34b82-4df3-44f5-a3f3-c8418b1ab601_isrForCrosstalkSources_1031713_16.stderr parsl.dataflow.dflow INFO: Task 2397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0739b62ab056bb4b6df964f5de19e42 with result from task 2397 parsl.dataflow.dflow DEBUG: Task 2397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b145da29-11b0-4cee-8035-16ad43371759_isrForCrosstalkSources_1031703_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b145da29-11b0-4cee-8035-16ad43371759_isrForCrosstalkSources_1031703_11.stderr parsl.dataflow.dflow INFO: Task 2361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 280d3550ade255762c60553977edc0de with result from task 2361 parsl.dataflow.dflow DEBUG: Task 2361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/58b316b2-5f37-4b9b-adb1-05e7cfe1c1a6_isrForCrosstalkSources_1031712_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/58b316b2-5f37-4b9b-adb1-05e7cfe1c1a6_isrForCrosstalkSources_1031712_2.stderr parsl.dataflow.dflow INFO: Task 2078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0027aa9de1153a1d90f542d54e05296d with result from task 2078 parsl.dataflow.dflow DEBUG: Task 2078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/a2205d95-88d3-4147-8454-95fd1708d097_isrForCrosstalkSources_1031714_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/a2205d95-88d3-4147-8454-95fd1708d097_isrForCrosstalkSources_1031714_60.stderr parsl.dataflow.dflow INFO: Task 2120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d94de5745de5059cc2b19e81d9a705c8 with result from task 2120 parsl.dataflow.dflow DEBUG: Task 2120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/094b63be-3559-450f-925a-03f891a67324_isrForCrosstalkSources_1031724_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/094b63be-3559-450f-925a-03f891a67324_isrForCrosstalkSources_1031724_10.stderr parsl.dataflow.dflow INFO: Task 2305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f451b9dbe7ad9f7cb6f867beb97b358e with result from task 2305 parsl.dataflow.dflow DEBUG: Task 2305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/a200cf15-86da-4dbc-ba08-a825d4654fef_isrForCrosstalkSources_1031703_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/a200cf15-86da-4dbc-ba08-a825d4654fef_isrForCrosstalkSources_1031703_15.stderr parsl.dataflow.dflow INFO: Task 2268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d45e525abbb51d2ec1938b654ab37787 with result from task 2268 parsl.dataflow.dflow DEBUG: Task 2268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0e7519fe-8016-4479-af2e-26eb37c8f9b2_isrForCrosstalkSources_1031706_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031706/0e7519fe-8016-4479-af2e-26eb37c8f9b2_isrForCrosstalkSources_1031706_50.stderr parsl.dataflow.dflow INFO: Task 1978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f0098cdc475fd90b4f0a5a949f355c9 with result from task 1978 parsl.dataflow.dflow DEBUG: Task 1978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/0c2329cb-7e1c-4682-84ea-8573e7f105a6_isrForCrosstalkSources_1031733_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/0c2329cb-7e1c-4682-84ea-8573e7f105a6_isrForCrosstalkSources_1031733_62.stderr parsl.dataflow.dflow INFO: Task 2254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 270373d679745aa907787803f5972966 with result from task 2254 parsl.dataflow.dflow DEBUG: Task 2254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/b68cf1f2-9be2-4a1f-94a3-1f8f1f13ac2c_isrForCrosstalkSources_1031733_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031733/b68cf1f2-9be2-4a1f-94a3-1f8f1f13ac2c_isrForCrosstalkSources_1031733_58.stderr parsl.dataflow.dflow INFO: Task 2102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be46127ff286ff65f69dab747035b715 with result from task 2102 parsl.dataflow.dflow DEBUG: Task 2102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/0ab90373-c1ef-4748-b96a-408c88e94118_isrForCrosstalkSources_1031730_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/0ab90373-c1ef-4748-b96a-408c88e94118_isrForCrosstalkSources_1031730_62.stderr parsl.dataflow.dflow INFO: Task 2267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e34172efbf2a6819e6d3cd607f403946 with result from task 2267 parsl.dataflow.dflow DEBUG: Task 2267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/034a4f57-b09b-418e-a5e9-7164e4355108_isrForCrosstalkSources_1031718_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/034a4f57-b09b-418e-a5e9-7164e4355108_isrForCrosstalkSources_1031718_59.stderr parsl.dataflow.dflow INFO: Task 2344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42b5cf05a5521be1652c8aa8bae83cde with result from task 2344 parsl.dataflow.dflow DEBUG: Task 2344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/33d8cfc1-1496-4320-a0c3-eb0b00a78542_isrForCrosstalkSources_1031721_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/33d8cfc1-1496-4320-a0c3-eb0b00a78542_isrForCrosstalkSources_1031721_30.stderr parsl.dataflow.dflow INFO: Task 2253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b7a45c32d8b985e385c459424a68aa7 with result from task 2253 parsl.dataflow.dflow DEBUG: Task 2253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/6dccfee7-cfc7-4c5f-9c8c-8c4a61eeba3e_isrForCrosstalkSources_1031720_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/6dccfee7-cfc7-4c5f-9c8c-8c4a61eeba3e_isrForCrosstalkSources_1031720_55.stderr parsl.dataflow.dflow INFO: Task 2442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 900aa97ada01945d7a69a9c654c4ac2e with result from task 2442 parsl.dataflow.dflow DEBUG: Task 2442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e828b06a-b768-40cf-9ac0-994a4a6c58d5_isrForCrosstalkSources_1031738_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/e828b06a-b768-40cf-9ac0-994a4a6c58d5_isrForCrosstalkSources_1031738_24.stderr parsl.dataflow.dflow INFO: Task 2113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91a371de952db301585fca17d7b3ceaa with result from task 2113 parsl.dataflow.dflow DEBUG: Task 2113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6da3e724-0cbd-4fc4-b9ad-71e5f5c19d41_isrForCrosstalkSources_1031723_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/6da3e724-0cbd-4fc4-b9ad-71e5f5c19d41_isrForCrosstalkSources_1031723_35.stderr parsl.dataflow.dflow INFO: Task 2458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f43ba138437218f70e04911a44c6df4e with result from task 2458 parsl.dataflow.dflow DEBUG: Task 2458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/637572c0-3333-45d1-966f-aa56107eb107_isrForCrosstalkSources_1031732_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031732/637572c0-3333-45d1-966f-aa56107eb107_isrForCrosstalkSources_1031732_28.stderr parsl.dataflow.dflow INFO: Task 2317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75c32fb73eff1b5da5dd16df1a747e4b with result from task 2317 parsl.dataflow.dflow DEBUG: Task 2317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/2f3e982f-0df0-4cf2-ab6f-4c2f735d9679_isrForCrosstalkSources_1031712_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/2f3e982f-0df0-4cf2-ab6f-4c2f735d9679_isrForCrosstalkSources_1031712_41.stderr parsl.dataflow.dflow INFO: Task 2376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 159407c3fea3a8072332076bb71e95da with result from task 2376 parsl.dataflow.dflow DEBUG: Task 2376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/b2c72f0c-ad7c-4170-bcf2-a47648abd309_isrForCrosstalkSources_1031709_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/b2c72f0c-ad7c-4170-bcf2-a47648abd309_isrForCrosstalkSources_1031709_30.stderr parsl.dataflow.dflow INFO: Task 2209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e73cd0d8b7ad10340ed5061829eece3 with result from task 2209 parsl.dataflow.dflow DEBUG: Task 2209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/dac518f5-c6ad-4481-b129-6b3424f05860_isrForCrosstalkSources_1031714_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/dac518f5-c6ad-4481-b129-6b3424f05860_isrForCrosstalkSources_1031714_31.stderr parsl.dataflow.dflow INFO: Task 2285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57e28d1048cb8fc3af25ba5f95507f21 with result from task 2285 parsl.dataflow.dflow DEBUG: Task 2285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/2a1b14a9-9389-4948-bf8e-fe4797a5336d_isrForCrosstalkSources_1031715_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/2a1b14a9-9389-4948-bf8e-fe4797a5336d_isrForCrosstalkSources_1031715_12.stderr parsl.dataflow.dflow INFO: Task 2370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4e42ad57dfd392c884eed3586268be9 with result from task 2370 parsl.dataflow.dflow DEBUG: Task 2370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/06f34c6b-ad02-4e4b-9297-205ecba7cc51_isrForCrosstalkSources_1031702_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/06f34c6b-ad02-4e4b-9297-205ecba7cc51_isrForCrosstalkSources_1031702_58.stderr parsl.dataflow.dflow INFO: Task 2300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8be1ecee1546b5a5e3b1923a91b93fff with result from task 2300 parsl.dataflow.dflow DEBUG: Task 2300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d7b7ffdd-da45-442a-91ba-1b17720e7c64_isrForCrosstalkSources_1031725_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/d7b7ffdd-da45-442a-91ba-1b17720e7c64_isrForCrosstalkSources_1031725_54.stderr parsl.dataflow.dflow INFO: Task 2426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88d21f97ccba3b9a5e87292452fa0383 with result from task 2426 parsl.dataflow.dflow DEBUG: Task 2426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/87b92d26-6127-4a7d-8c96-edbdbaef4b45_isrForCrosstalkSources_1031718_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031718/87b92d26-6127-4a7d-8c96-edbdbaef4b45_isrForCrosstalkSources_1031718_7.stderr parsl.dataflow.dflow INFO: Task 2237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4edbfe4e42df0f97d9ddbd717ae5e675 with result from task 2237 parsl.dataflow.dflow DEBUG: Task 2237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bc1842bf-2c7f-42d1-806b-30395ec6c657_isrForCrosstalkSources_1031715_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/bc1842bf-2c7f-42d1-806b-30395ec6c657_isrForCrosstalkSources_1031715_30.stderr parsl.dataflow.dflow INFO: Task 2232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 792422fbd16892cb0f7751c47d8d4c3f with result from task 2232 parsl.dataflow.dflow DEBUG: Task 2232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d8328b06-d650-415c-bab0-910439a511e9_isrForCrosstalkSources_1031715_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031715/d8328b06-d650-415c-bab0-910439a511e9_isrForCrosstalkSources_1031715_48.stderr parsl.dataflow.dflow INFO: Task 2219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b4b83fba81edf0067c792ee070629e5 with result from task 2219 parsl.dataflow.dflow DEBUG: Task 2219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/69b896c2-cb76-4088-858f-75ba91c1f86a_isrForCrosstalkSources_1031727_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/69b896c2-cb76-4088-858f-75ba91c1f86a_isrForCrosstalkSources_1031727_28.stderr parsl.dataflow.dflow INFO: Task 2337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d61655169242869dd268caa99fd3f05a with result from task 2337 parsl.dataflow.dflow DEBUG: Task 2337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/1ad69b10-7057-4641-88fb-90ac02daec38_isrForCrosstalkSources_1031722_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/1ad69b10-7057-4641-88fb-90ac02daec38_isrForCrosstalkSources_1031722_53.stderr parsl.dataflow.dflow INFO: Task 2429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e18853b03c18c1622392602e1780ec6c with result from task 2429 parsl.dataflow.dflow DEBUG: Task 2429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/cd8f8d7f-1f3f-4e96-8ba9-add12d0f2a80_isrForCrosstalkSources_1031735_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/cd8f8d7f-1f3f-4e96-8ba9-add12d0f2a80_isrForCrosstalkSources_1031735_37.stderr parsl.dataflow.dflow INFO: Task 2438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 648698f541d3c1aefb00f726bf828ba0 with result from task 2438 parsl.dataflow.dflow DEBUG: Task 2438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/1f060d83-339a-4775-a160-ce248f7243d0_isrForCrosstalkSources_1031736_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/1f060d83-339a-4775-a160-ce248f7243d0_isrForCrosstalkSources_1031736_37.stderr parsl.dataflow.dflow INFO: Task 2346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 368af57b16c2043837703512f8748062 with result from task 2346 parsl.dataflow.dflow DEBUG: Task 2346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/9e89c959-39b4-4578-90af-13d831325270_isrForCrosstalkSources_1031722_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/9e89c959-39b4-4578-90af-13d831325270_isrForCrosstalkSources_1031722_28.stderr parsl.dataflow.dflow INFO: Task 2398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea64835b4f43d2c6018da152db111a25 with result from task 2398 parsl.dataflow.dflow DEBUG: Task 2398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/08e101bc-16c2-41e7-89a8-5c831a27cfbc_isrForCrosstalkSources_1031712_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031712/08e101bc-16c2-41e7-89a8-5c831a27cfbc_isrForCrosstalkSources_1031712_36.stderr parsl.dataflow.dflow INFO: Task 2463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a8375a2d5bb3b5538c92f6ae8b644b5 with result from task 2463 parsl.dataflow.dflow DEBUG: Task 2463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/93257301-af67-4b6e-8202-fa5acd18042f_isrForCrosstalkSources_1031730_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/93257301-af67-4b6e-8202-fa5acd18042f_isrForCrosstalkSources_1031730_47.stderr parsl.dataflow.dflow INFO: Task 2410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a236b2d1ee3c01f75bc0beaa73efedc with result from task 2410 parsl.dataflow.dflow DEBUG: Task 2410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/69041da3-fe9f-42d1-b5f0-fff958c02876_isrForCrosstalkSources_1031735_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031735/69041da3-fe9f-42d1-b5f0-fff958c02876_isrForCrosstalkSources_1031735_60.stderr parsl.dataflow.dflow INFO: Task 2334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b6f999c5aca3cae4fa862cf2d4c46a3 with result from task 2334 parsl.dataflow.dflow DEBUG: Task 2334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/76ded7eb-e838-4816-83d0-8c7f3f159d70_isrForCrosstalkSources_1031721_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/76ded7eb-e838-4816-83d0-8c7f3f159d70_isrForCrosstalkSources_1031721_59.stderr parsl.dataflow.dflow INFO: Task 2112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6e7c1386e05e8a317d931821404e2ac with result from task 2112 parsl.dataflow.dflow DEBUG: Task 2112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c4313041-7ee8-49d2-b1cb-07d3a8756cb7_isrForCrosstalkSources_1031731_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/c4313041-7ee8-49d2-b1cb-07d3a8756cb7_isrForCrosstalkSources_1031731_29.stderr parsl.dataflow.dflow INFO: Task 2116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba4148d2fff30c90c6856a1a4e6ac760 with result from task 2116 parsl.dataflow.dflow DEBUG: Task 2116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5cd3bd48-443b-4ffa-93ab-6eb9a8c1dbad_isrForCrosstalkSources_1031728_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/5cd3bd48-443b-4ffa-93ab-6eb9a8c1dbad_isrForCrosstalkSources_1031728_43.stderr parsl.dataflow.dflow INFO: Task 2448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58948f6475a908a8b7e1d500a56a6f7c with result from task 2448 parsl.dataflow.dflow DEBUG: Task 2448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e536eddc-1da3-42b8-8c59-0c1a31fb0996_isrForCrosstalkSources_1031720_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/e536eddc-1da3-42b8-8c59-0c1a31fb0996_isrForCrosstalkSources_1031720_48.stderr parsl.dataflow.dflow INFO: Task 2466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5770c0573d1c71831d09fed730b20f9f with result from task 2466 parsl.dataflow.dflow DEBUG: Task 2466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a5537f65-d1a5-4154-b8bb-5eb463d9d4fe_isrForCrosstalkSources_1031725_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/a5537f65-d1a5-4154-b8bb-5eb463d9d4fe_isrForCrosstalkSources_1031725_53.stderr parsl.dataflow.dflow INFO: Task 2244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 670a02b28b59265e48de4d8bf686f35f with result from task 2244 parsl.dataflow.dflow DEBUG: Task 2244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/51ec72ff-a78c-4261-876f-3bf3ea59aa07_isrForCrosstalkSources_1031730_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031730/51ec72ff-a78c-4261-876f-3bf3ea59aa07_isrForCrosstalkSources_1031730_37.stderr parsl.dataflow.dflow INFO: Task 2306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae3f1472067103ed6a5c1d4b09cc0c13 with result from task 2306 parsl.dataflow.dflow DEBUG: Task 2306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/513208a5-6715-4e36-b5aa-f16ab42bec37_isrForCrosstalkSources_1031726_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/513208a5-6715-4e36-b5aa-f16ab42bec37_isrForCrosstalkSources_1031726_9.stderr parsl.dataflow.dflow INFO: Task 2440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80944c3d01cd0e517eff11b8dc3ec469 with result from task 2440 parsl.dataflow.dflow DEBUG: Task 2440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1e154b85-4c07-4995-a7e6-28898078b2d4_isrForCrosstalkSources_1031700_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/1e154b85-4c07-4995-a7e6-28898078b2d4_isrForCrosstalkSources_1031700_30.stderr parsl.dataflow.dflow INFO: Task 2338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83aff81d0cf8543d7437e82170c761dd with result from task 2338 parsl.dataflow.dflow DEBUG: Task 2338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/16bb19ec-6b9f-427c-8e22-4774cb9d6a62_isrForCrosstalkSources_1031723_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/16bb19ec-6b9f-427c-8e22-4774cb9d6a62_isrForCrosstalkSources_1031723_31.stderr parsl.dataflow.dflow INFO: Task 2343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ac09df2ed9ee1c711dfffbef02939ce with result from task 2343 parsl.dataflow.dflow DEBUG: Task 2343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/6377f5b2-aab4-4dc6-ba03-212c79047005_isrForCrosstalkSources_1031708_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/6377f5b2-aab4-4dc6-ba03-212c79047005_isrForCrosstalkSources_1031708_52.stderr parsl.dataflow.dflow INFO: Task 2436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4b189a15a91dc2481c29d6765087e40 with result from task 2436 parsl.dataflow.dflow DEBUG: Task 2436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7ce6cefb-b6b2-4867-8000-74e7f8235c21_isrForCrosstalkSources_1031702_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/7ce6cefb-b6b2-4867-8000-74e7f8235c21_isrForCrosstalkSources_1031702_47.stderr parsl.dataflow.dflow INFO: Task 2393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5a750eae6c826b1d375fdbb3f25d37f with result from task 2393 parsl.dataflow.dflow DEBUG: Task 2393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/89daeb8d-83dc-4a46-8f64-1fb68609cc9b_isrForCrosstalkSources_1031738_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/89daeb8d-83dc-4a46-8f64-1fb68609cc9b_isrForCrosstalkSources_1031738_43.stderr parsl.dataflow.dflow INFO: Task 2455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fb4262b36e0384c514f83b0835761e0 with result from task 2455 parsl.dataflow.dflow DEBUG: Task 2455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/1bfa01c3-aa56-4956-bcd6-072544c494ef_isrForCrosstalkSources_1031738_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/1bfa01c3-aa56-4956-bcd6-072544c494ef_isrForCrosstalkSources_1031738_54.stderr parsl.dataflow.dflow INFO: Task 2375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf46c3884bfd1736cdead5daa3ff64c4 with result from task 2375 parsl.dataflow.dflow DEBUG: Task 2375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7a28086c-dca6-4861-ba1e-0d7f18d9db8a_isrForCrosstalkSources_1031725_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/7a28086c-dca6-4861-ba1e-0d7f18d9db8a_isrForCrosstalkSources_1031725_46.stderr parsl.dataflow.dflow INFO: Task 2421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 702c8caeaf53bc369a016a38525cafac with result from task 2421 parsl.dataflow.dflow DEBUG: Task 2421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/7181138f-8ddf-4a45-a2e0-e7904a3d2cf7_isrForCrosstalkSources_1031724_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/7181138f-8ddf-4a45-a2e0-e7904a3d2cf7_isrForCrosstalkSources_1031724_32.stderr parsl.dataflow.dflow INFO: Task 2303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 735c56c72887b15f70622a0733c09b2a with result from task 2303 parsl.dataflow.dflow DEBUG: Task 2303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/da7d8998-e87e-4f54-9c27-434f14e3a322_isrForCrosstalkSources_1031704_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/da7d8998-e87e-4f54-9c27-434f14e3a322_isrForCrosstalkSources_1031704_44.stderr parsl.dataflow.dflow INFO: Task 2454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a153eacfafd1c341adc049601e0f29d with result from task 2454 parsl.dataflow.dflow DEBUG: Task 2454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/4c19818d-9146-49b4-9237-808a9b14d1ee_isrForCrosstalkSources_1031724_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031724/4c19818d-9146-49b4-9237-808a9b14d1ee_isrForCrosstalkSources_1031724_15.stderr parsl.dataflow.dflow INFO: Task 2350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a53587378eec7577dc1e2407ea8a56f with result from task 2350 parsl.dataflow.dflow DEBUG: Task 2350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8c1ae541-3c12-4459-9002-cbd8be159b7a_isrForCrosstalkSources_1031709_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/8c1ae541-3c12-4459-9002-cbd8be159b7a_isrForCrosstalkSources_1031709_47.stderr parsl.dataflow.dflow INFO: Task 2404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b4548379c396bbec58e18205bb4f171 with result from task 2404 parsl.dataflow.dflow DEBUG: Task 2404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1dd17c85-50dc-4e77-bf95-9f4d94d24b3f_isrForCrosstalkSources_1031709_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/1dd17c85-50dc-4e77-bf95-9f4d94d24b3f_isrForCrosstalkSources_1031709_3.stderr 2024-03-17 04:07:30 proc_lsst.multi:170 [INFO] [multi] job_id 17148217 proc_lsst.multi INFO: [multi] job_id 17148217 2024-03-17 04:07:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 214 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 214 2024-03-17 04:07:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148217 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148217 parsl.executors.status_handling DEBUG: Launched block 270 on executor multi with job ID 17148217 parsl.executors.status_handling INFO: Allocated block ID 271 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-271 2024-03-17 04:07:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-271 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-271 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 214 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 214 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c914ea070d13029747a01bd673e5dab with result from task 2399 parsl.dataflow.dflow DEBUG: Task 2399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b7fd96c3-debc-4c52-9ab4-b13e38ef7c39_isrForCrosstalkSources_1031714_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/b7fd96c3-debc-4c52-9ab4-b13e38ef7c39_isrForCrosstalkSources_1031714_36.stderr 2024-03-17 04:07:30 proc_lsst.multi:170 [INFO] [multi] job_id 17148218 proc_lsst.multi INFO: [multi] job_id 17148218 2024-03-17 04:07:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 215 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 215 2024-03-17 04:07:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148218 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148218 parsl.executors.status_handling DEBUG: Launched block 271 on executor multi with job ID 17148218 parsl.executors.status_handling INFO: Allocated block ID 272 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-272 2024-03-17 04:07:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-272 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-272 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 215 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 215 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5e5320be33f23a20868c4d758c7e986 with result from task 2412 parsl.dataflow.dflow DEBUG: Task 2412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0f7a6ccc-f11a-4ded-a84c-005a83f8cfb3_isrForCrosstalkSources_1031707_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031707/0f7a6ccc-f11a-4ded-a84c-005a83f8cfb3_isrForCrosstalkSources_1031707_29.stderr 2024-03-17 04:07:30 proc_lsst.multi:170 [INFO] [multi] job_id 17148219 proc_lsst.multi INFO: [multi] job_id 17148219 2024-03-17 04:07:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 216 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 216 2024-03-17 04:07:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148219 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148219 parsl.executors.status_handling DEBUG: Launched block 272 on executor multi with job ID 17148219 parsl.executors.status_handling INFO: Allocated block ID 273 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-273 2024-03-17 04:07:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-273 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-273 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 216 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 216 2024-03-17 04:07:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b1a83ac2033911b5165f3637ff2ed62 with result from task 2378 parsl.dataflow.dflow DEBUG: Task 2378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/658810b7-189f-45d5-b6d3-46d560d7f003_isrForCrosstalkSources_1031719_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031719/658810b7-189f-45d5-b6d3-46d560d7f003_isrForCrosstalkSources_1031719_31.stderr 2024-03-17 04:07:31 proc_lsst.multi:170 [INFO] [multi] job_id 17148220 proc_lsst.multi INFO: [multi] job_id 17148220 2024-03-17 04:07:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 217 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 217 2024-03-17 04:07:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148220 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148220 parsl.executors.status_handling DEBUG: Launched block 273 on executor multi with job ID 17148220 parsl.executors.status_handling INFO: Allocated block ID 274 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-274 2024-03-17 04:07:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-274 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-274 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 217 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 217 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry faf90b15c5cc5a22a6939a519244bc14 with result from task 2419 parsl.dataflow.dflow DEBUG: Task 2419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/11f62f69-4899-4ccf-9a12-dfe22a4cf8b9_isrForCrosstalkSources_1031703_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/11f62f69-4899-4ccf-9a12-dfe22a4cf8b9_isrForCrosstalkSources_1031703_36.stderr parsl.dataflow.dflow INFO: Task 2355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57de53f76c8fc562d18c7ee5e21dbfd1 with result from task 2355 parsl.dataflow.dflow DEBUG: Task 2355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/418c6006-20c2-4203-9125-52cb1d04ce56_isrForCrosstalkSources_1031701_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/418c6006-20c2-4203-9125-52cb1d04ce56_isrForCrosstalkSources_1031701_57.stderr 2024-03-17 04:07:31 proc_lsst.multi:170 [INFO] [multi] job_id 17148221 proc_lsst.multi INFO: [multi] job_id 17148221 2024-03-17 04:07:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 218 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 218 2024-03-17 04:07:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148221 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148221 parsl.executors.status_handling DEBUG: Launched block 274 on executor multi with job ID 17148221 parsl.executors.status_handling INFO: Allocated block ID 275 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-275 2024-03-17 04:07:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-275 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-275 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 218 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 218 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:31 proc_lsst.multi:170 [INFO] [multi] job_id 17148222 proc_lsst.multi INFO: [multi] job_id 17148222 2024-03-17 04:07:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 219 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 219 2024-03-17 04:07:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148222 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148222 parsl.executors.status_handling DEBUG: Launched block 275 on executor multi with job ID 17148222 parsl.executors.status_handling INFO: Allocated block ID 276 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-276 2024-03-17 04:07:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-276 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-276 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 219 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 219 2024-03-17 04:07:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73bcffdca59b5ed814fe40f71cbd1805 with result from task 2414 parsl.dataflow.dflow DEBUG: Task 2414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/ebd03b2a-bedf-48e9-88fb-988bd35b37ab_isrForCrosstalkSources_1031704_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031704/ebd03b2a-bedf-48e9-88fb-988bd35b37ab_isrForCrosstalkSources_1031704_62.stderr parsl.dataflow.dflow INFO: Task 2459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae54bbbabb58e0e2fdf504db2d0e8072 with result from task 2459 parsl.dataflow.dflow DEBUG: Task 2459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b576cf42-f0b0-4525-a1b6-6ab879563651_isrForCrosstalkSources_1031708_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031708/b576cf42-f0b0-4525-a1b6-6ab879563651_isrForCrosstalkSources_1031708_24.stderr 2024-03-17 04:07:32 proc_lsst.multi:170 [INFO] [multi] job_id 17148223 proc_lsst.multi INFO: [multi] job_id 17148223 2024-03-17 04:07:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 220 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 220 2024-03-17 04:07:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148223 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148223 parsl.executors.status_handling DEBUG: Launched block 276 on executor multi with job ID 17148223 parsl.executors.status_handling INFO: Allocated block ID 277 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-277 2024-03-17 04:07:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-277 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-277 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 220 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 220 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f558c607cf60e05a85aea23d21c47102 with result from task 2472 parsl.dataflow.dflow DEBUG: Task 2472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/128cf33d-641a-4430-9c39-7186b5b53e5d_isrForCrosstalkSources_1031729_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031729/128cf33d-641a-4430-9c39-7186b5b53e5d_isrForCrosstalkSources_1031729_13.stderr 2024-03-17 04:07:32 proc_lsst.multi:170 [INFO] [multi] job_id 17148224 proc_lsst.multi INFO: [multi] job_id 17148224 2024-03-17 04:07:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 221 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 221 2024-03-17 04:07:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148224 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148224 parsl.executors.status_handling DEBUG: Launched block 277 on executor multi with job ID 17148224 parsl.executors.status_handling INFO: Allocated block ID 278 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-278 2024-03-17 04:07:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-278 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-278 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 221 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 221 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:32 proc_lsst.multi:170 [INFO] [multi] job_id 17148225 proc_lsst.multi INFO: [multi] job_id 17148225 2024-03-17 04:07:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 222 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 222 2024-03-17 04:07:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148225 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148225 parsl.executors.status_handling DEBUG: Launched block 278 on executor multi with job ID 17148225 parsl.executors.status_handling INFO: Allocated block ID 279 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-279 2024-03-17 04:07:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-279 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-279 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 222 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 222 2024-03-17 04:07:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a798c50bf0dfdd7ab86d58485d6dfeb1 with result from task 2428 parsl.dataflow.dflow DEBUG: Task 2428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/baa393a5-a35b-4a11-800b-e6af21968eab_isrForCrosstalkSources_1031731_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/baa393a5-a35b-4a11-800b-e6af21968eab_isrForCrosstalkSources_1031731_40.stderr parsl.dataflow.dflow INFO: Task 2475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5bd9577379b37f8598c352b34c2d7d9 with result from task 2475 parsl.dataflow.dflow DEBUG: Task 2475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a9d0fd0e-557f-43c3-a9e7-bced034d41a5_isrForCrosstalkSources_1031709_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031709/a9d0fd0e-557f-43c3-a9e7-bced034d41a5_isrForCrosstalkSources_1031709_22.stderr parsl.dataflow.dflow INFO: Task 2471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4becee01fd2dedc7265339b192b28dec with result from task 2471 parsl.dataflow.dflow DEBUG: Task 2471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d142063e-daf5-47d6-acec-dd125a8980fb_isrForCrosstalkSources_1031722_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031722/d142063e-daf5-47d6-acec-dd125a8980fb_isrForCrosstalkSources_1031722_7.stderr 2024-03-17 04:07:33 proc_lsst.multi:170 [INFO] [multi] job_id 17148226 proc_lsst.multi INFO: [multi] job_id 17148226 2024-03-17 04:07:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 223 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 223 2024-03-17 04:07:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148226 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148226 parsl.executors.status_handling DEBUG: Launched block 279 on executor multi with job ID 17148226 parsl.executors.status_handling INFO: Allocated block ID 280 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-280 2024-03-17 04:07:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-280 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-280 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 223 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 223 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:33 proc_lsst.multi:170 [INFO] [multi] job_id 17148227 proc_lsst.multi INFO: [multi] job_id 17148227 2024-03-17 04:07:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 224 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 224 2024-03-17 04:07:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148227 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148227 parsl.executors.status_handling DEBUG: Launched block 280 on executor multi with job ID 17148227 parsl.executors.status_handling INFO: Allocated block ID 281 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-281 2024-03-17 04:07:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-281 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-281 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 224 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 224 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5153c5b371853d1d5487915d9e2d57fc with result from task 2473 parsl.dataflow.dflow DEBUG: Task 2473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bc83decb-dd28-4301-abad-3dc5db9aa7ac_isrForCrosstalkSources_1031723_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031723/bc83decb-dd28-4301-abad-3dc5db9aa7ac_isrForCrosstalkSources_1031723_60.stderr parsl.dataflow.dflow INFO: Task 2470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7d177b496183c5396eabea0b159b6cd with result from task 2470 parsl.dataflow.dflow DEBUG: Task 2470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/16acf4fd-65d7-4626-bceb-7c3aaa476ea8_isrForCrosstalkSources_1031736_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031736/16acf4fd-65d7-4626-bceb-7c3aaa476ea8_isrForCrosstalkSources_1031736_61.stderr 2024-03-17 04:07:33 proc_lsst.multi:170 [INFO] [multi] job_id 17148228 proc_lsst.multi INFO: [multi] job_id 17148228 2024-03-17 04:07:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 225 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 225 2024-03-17 04:07:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148228 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148228 parsl.executors.status_handling DEBUG: Launched block 281 on executor multi with job ID 17148228 parsl.executors.status_handling INFO: Allocated block ID 282 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-282 2024-03-17 04:07:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-282 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-282 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 225 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 225 2024-03-17 04:07:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bd53ce6682946992ff38e186f2efe8c with result from task 2441 parsl.dataflow.dflow DEBUG: Task 2441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1d7ce7a5-45a9-40f7-844c-011e2e8937c7_isrForCrosstalkSources_1031703_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/1d7ce7a5-45a9-40f7-844c-011e2e8937c7_isrForCrosstalkSources_1031703_58.stderr 2024-03-17 04:07:34 proc_lsst.multi:170 [INFO] [multi] job_id 17148229 proc_lsst.multi INFO: [multi] job_id 17148229 2024-03-17 04:07:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 226 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 226 2024-03-17 04:07:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148229 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148229 parsl.executors.status_handling DEBUG: Launched block 282 on executor multi with job ID 17148229 parsl.executors.status_handling INFO: Allocated block ID 283 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-283 2024-03-17 04:07:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-283 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-283 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 226 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 226 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:34 proc_lsst.multi:170 [INFO] [multi] job_id 17148230 proc_lsst.multi INFO: [multi] job_id 17148230 2024-03-17 04:07:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 227 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 227 2024-03-17 04:07:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148230 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148230 parsl.executors.status_handling DEBUG: Launched block 283 on executor multi with job ID 17148230 parsl.executors.status_handling INFO: Allocated block ID 284 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-284 2024-03-17 04:07:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-284 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-284 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 227 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 227 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:34 proc_lsst.multi:170 [INFO] [multi] job_id 17148231 proc_lsst.multi INFO: [multi] job_id 17148231 2024-03-17 04:07:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 228 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 228 2024-03-17 04:07:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148231 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148231 parsl.executors.status_handling DEBUG: Launched block 284 on executor multi with job ID 17148231 parsl.executors.status_handling INFO: Allocated block ID 285 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-285 2024-03-17 04:07:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-285 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-285 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 228 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 228 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:34 proc_lsst.multi:170 [INFO] [multi] job_id 17148232 proc_lsst.multi INFO: [multi] job_id 17148232 2024-03-17 04:07:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 229 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 229 2024-03-17 04:07:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148232 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148232 parsl.executors.status_handling DEBUG: Launched block 285 on executor multi with job ID 17148232 parsl.executors.status_handling INFO: Allocated block ID 286 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-286 2024-03-17 04:07:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-286 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-286 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 229 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 229 2024-03-17 04:07:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:35 proc_lsst.multi:170 [INFO] [multi] job_id 17148233 proc_lsst.multi INFO: [multi] job_id 17148233 2024-03-17 04:07:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 230 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 230 2024-03-17 04:07:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148233 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148233 parsl.executors.status_handling DEBUG: Launched block 286 on executor multi with job ID 17148233 parsl.executors.status_handling INFO: Allocated block ID 287 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-287 2024-03-17 04:07:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-287 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-287 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 230 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 230 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:35 proc_lsst.multi:170 [INFO] [multi] job_id 17148234 proc_lsst.multi INFO: [multi] job_id 17148234 2024-03-17 04:07:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 231 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 231 2024-03-17 04:07:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148234 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148234 parsl.executors.status_handling DEBUG: Launched block 287 on executor multi with job ID 17148234 parsl.executors.status_handling INFO: Allocated block ID 288 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-288 2024-03-17 04:07:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-288 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-288 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 231 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 231 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:35 proc_lsst.multi:170 [INFO] [multi] job_id 17148235 proc_lsst.multi INFO: [multi] job_id 17148235 2024-03-17 04:07:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 232 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 232 2024-03-17 04:07:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148235 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148235 parsl.executors.status_handling DEBUG: Launched block 288 on executor multi with job ID 17148235 parsl.executors.status_handling INFO: Allocated block ID 289 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-289 2024-03-17 04:07:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-289 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-289 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 232 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 232 2024-03-17 04:07:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:36 proc_lsst.multi:170 [INFO] [multi] job_id 17148236 proc_lsst.multi INFO: [multi] job_id 17148236 2024-03-17 04:07:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 233 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 233 2024-03-17 04:07:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148236 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148236 parsl.executors.status_handling DEBUG: Launched block 289 on executor multi with job ID 17148236 parsl.executors.status_handling INFO: Allocated block ID 290 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-290 2024-03-17 04:07:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-290 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-290 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 233 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 233 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:36 proc_lsst.multi:170 [INFO] [multi] job_id 17148237 proc_lsst.multi INFO: [multi] job_id 17148237 2024-03-17 04:07:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 234 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 234 2024-03-17 04:07:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148237 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148237 parsl.executors.status_handling DEBUG: Launched block 290 on executor multi with job ID 17148237 parsl.executors.status_handling INFO: Allocated block ID 291 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-291 2024-03-17 04:07:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-291 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-291 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 234 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 234 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:36 proc_lsst.multi:170 [INFO] [multi] job_id 17148238 proc_lsst.multi INFO: [multi] job_id 17148238 2024-03-17 04:07:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 235 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 235 2024-03-17 04:07:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148238 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148238 parsl.executors.status_handling DEBUG: Launched block 291 on executor multi with job ID 17148238 parsl.executors.status_handling INFO: Allocated block ID 292 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-292 2024-03-17 04:07:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-292 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-292 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 235 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 235 2024-03-17 04:07:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:37 proc_lsst.multi:170 [INFO] [multi] job_id 17148239 proc_lsst.multi INFO: [multi] job_id 17148239 2024-03-17 04:07:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 236 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 236 2024-03-17 04:07:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148239 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148239 parsl.executors.status_handling DEBUG: Launched block 292 on executor multi with job ID 17148239 parsl.executors.status_handling INFO: Allocated block ID 293 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-293 2024-03-17 04:07:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-293 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-293 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 236 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 236 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:37 proc_lsst.multi:170 [INFO] [multi] job_id 17148240 proc_lsst.multi INFO: [multi] job_id 17148240 2024-03-17 04:07:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 237 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 237 2024-03-17 04:07:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148240 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148240 parsl.executors.status_handling DEBUG: Launched block 293 on executor multi with job ID 17148240 parsl.executors.status_handling INFO: Allocated block ID 294 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-294 2024-03-17 04:07:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-294 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-294 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 237 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 237 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:37 proc_lsst.multi:170 [INFO] [multi] job_id 17148241 proc_lsst.multi INFO: [multi] job_id 17148241 2024-03-17 04:07:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 238 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 238 2024-03-17 04:07:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148241 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148241 parsl.executors.status_handling DEBUG: Launched block 294 on executor multi with job ID 17148241 parsl.executors.status_handling INFO: Allocated block ID 295 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-295 2024-03-17 04:07:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-295 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-295 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 238 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 238 2024-03-17 04:07:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:38 proc_lsst.multi:170 [INFO] [multi] job_id 17148242 proc_lsst.multi INFO: [multi] job_id 17148242 2024-03-17 04:07:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 239 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 239 2024-03-17 04:07:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148242 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148242 parsl.executors.status_handling DEBUG: Launched block 295 on executor multi with job ID 17148242 parsl.executors.status_handling INFO: Allocated block ID 296 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-296 2024-03-17 04:07:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-296 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-296 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 239 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 239 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:38 proc_lsst.multi:170 [INFO] [multi] job_id 17148243 proc_lsst.multi INFO: [multi] job_id 17148243 2024-03-17 04:07:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 240 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 240 2024-03-17 04:07:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148243 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148243 parsl.executors.status_handling DEBUG: Launched block 296 on executor multi with job ID 17148243 parsl.executors.status_handling INFO: Allocated block ID 297 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-297 2024-03-17 04:07:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-297 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-297 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 240 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 240 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:38 proc_lsst.multi:170 [INFO] [multi] job_id 17148244 proc_lsst.multi INFO: [multi] job_id 17148244 2024-03-17 04:07:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 241 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 241 2024-03-17 04:07:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148244 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148244 parsl.executors.status_handling DEBUG: Launched block 297 on executor multi with job ID 17148244 parsl.executors.status_handling INFO: Allocated block ID 298 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-298 2024-03-17 04:07:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-298 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-298 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 241 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 241 2024-03-17 04:07:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33fea7e52817a3635224488cd5315121 with result from task 2034 parsl.dataflow.dflow DEBUG: Task 2034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/4ce6dbe9-ecd9-4fc2-8230-6ca12b917c56_isrForCrosstalkSources_1031710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/4ce6dbe9-ecd9-4fc2-8230-6ca12b917c56_isrForCrosstalkSources_1031710_2.stderr parsl.dataflow.dflow INFO: Task 2030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79383474c315f438a20a2057064b2cf2 with result from task 2030 parsl.dataflow.dflow DEBUG: Task 2030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/15c78d43-6e16-4a74-8a94-3f7d1053c6bd_isrForCrosstalkSources_1031721_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031721/15c78d43-6e16-4a74-8a94-3f7d1053c6bd_isrForCrosstalkSources_1031721_27.stderr parsl.dataflow.dflow INFO: Task 2032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f1e4809f0e3be442377fa3e8009f3c1 with result from task 2032 parsl.dataflow.dflow DEBUG: Task 2032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0e1473cc-5d94-4c17-bfa1-6beadb7219aa_isrForCrosstalkSources_1031720_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/0e1473cc-5d94-4c17-bfa1-6beadb7219aa_isrForCrosstalkSources_1031720_33.stderr parsl.dataflow.dflow INFO: Task 2036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71daa89d8bda89fd30b622f43d5202bd with result from task 2036 parsl.dataflow.dflow DEBUG: Task 2036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5975855d-a01d-4385-b562-b720038d9e39_isrForCrosstalkSources_1031727_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/5975855d-a01d-4385-b562-b720038d9e39_isrForCrosstalkSources_1031727_37.stderr parsl.dataflow.dflow INFO: Task 2029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 170646488cd38d3f8f5cf08031c6b4a3 with result from task 2029 parsl.dataflow.dflow DEBUG: Task 2029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a5cb0be1-fec7-4f5d-aecb-d946517cff4f_isrForCrosstalkSources_1031702_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031702/a5cb0be1-fec7-4f5d-aecb-d946517cff4f_isrForCrosstalkSources_1031702_31.stderr parsl.dataflow.dflow INFO: Task 2035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 734369f37f0b0f18a09692ebb59bdf02 with result from task 2035 parsl.dataflow.dflow DEBUG: Task 2035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d4a9a63b-e80c-43b4-b852-43c5b5503ca4_isrForCrosstalkSources_1031714_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/d4a9a63b-e80c-43b4-b852-43c5b5503ca4_isrForCrosstalkSources_1031714_49.stderr parsl.dataflow.dflow INFO: Task 2031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54fa7ce0934262dd6f142fa4ed731733 with result from task 2031 parsl.dataflow.dflow DEBUG: Task 2031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/eaf76b80-ae3b-4a38-8470-109be1cbc867_isrForCrosstalkSources_1031700_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031700/eaf76b80-ae3b-4a38-8470-109be1cbc867_isrForCrosstalkSources_1031700_8.stderr parsl.dataflow.dflow INFO: Task 2033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac287a4139d358c98084a2d9f4b4b0c5 with result from task 2033 parsl.dataflow.dflow DEBUG: Task 2033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a1836b94-487a-421b-b037-827a89108969_isrForCrosstalkSources_1031738_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031738/a1836b94-487a-421b-b037-827a89108969_isrForCrosstalkSources_1031738_57.stderr parsl.dataflow.dflow INFO: Task 2043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea7e086f7f964cae67a4c8ba9661026e with result from task 2043 parsl.dataflow.dflow DEBUG: Task 2043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/38be037e-c444-47a9-b3e1-bab212e3fd38_isrForCrosstalkSources_1031714_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031714/38be037e-c444-47a9-b3e1-bab212e3fd38_isrForCrosstalkSources_1031714_2.stderr parsl.dataflow.dflow INFO: Task 2040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e99ec2557d735b162ab8bb79cbf82b09 with result from task 2040 parsl.dataflow.dflow DEBUG: Task 2040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/06683ed6-ecac-49c2-9cd2-8b3576248c71_isrForCrosstalkSources_1031725_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031725/06683ed6-ecac-49c2-9cd2-8b3576248c71_isrForCrosstalkSources_1031725_33.stderr parsl.dataflow.dflow INFO: Task 2038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbcd0ce31d93bf5e2ac68cde59c2f3f5 with result from task 2038 parsl.dataflow.dflow DEBUG: Task 2038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a06fe411-639a-4a72-9752-f747991a86c8_isrForCrosstalkSources_1031713_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031713/a06fe411-639a-4a72-9752-f747991a86c8_isrForCrosstalkSources_1031713_50.stderr parsl.dataflow.dflow INFO: Task 2039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry daa00a87a706c15199ad45f41a1163c7 with result from task 2039 parsl.dataflow.dflow DEBUG: Task 2039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b1d97f9b-7633-4a04-a2cf-3c2f12228a34_isrForCrosstalkSources_1031703_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031703/b1d97f9b-7633-4a04-a2cf-3c2f12228a34_isrForCrosstalkSources_1031703_6.stderr parsl.dataflow.dflow INFO: Task 2050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91175689f458be953d25bddabb2af63e with result from task 2050 parsl.dataflow.dflow DEBUG: Task 2050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7ddf7117-820c-4400-b34d-e1ae6f3d9a7e_isrForCrosstalkSources_1031710_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031710/7ddf7117-820c-4400-b34d-e1ae6f3d9a7e_isrForCrosstalkSources_1031710_52.stderr parsl.dataflow.dflow INFO: Task 2048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1a7669e39a652a4ae196732ab414fbf with result from task 2048 parsl.dataflow.dflow DEBUG: Task 2048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/b4dfa5d8-882b-4416-83e6-bfbbb73719bc_isrForCrosstalkSources_1031717_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031717/b4dfa5d8-882b-4416-83e6-bfbbb73719bc_isrForCrosstalkSources_1031717_22.stderr parsl.dataflow.dflow INFO: Task 2051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67ad515a0889af877c679dc109808292 with result from task 2051 parsl.dataflow.dflow DEBUG: Task 2051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a93108e9-b789-4a8d-8611-2b481e7cc60e_isrForCrosstalkSources_1031720_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/a93108e9-b789-4a8d-8611-2b481e7cc60e_isrForCrosstalkSources_1031720_54.stderr parsl.dataflow.dflow INFO: Task 2049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a465bfdad9e6747cc14965ce5fbfe4ea with result from task 2049 parsl.dataflow.dflow DEBUG: Task 2049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/4d12196e-85da-461b-a5e3-a674c45f29b2_isrForCrosstalkSources_1031728_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031728/4d12196e-85da-461b-a5e3-a674c45f29b2_isrForCrosstalkSources_1031728_17.stderr parsl.dataflow.dflow INFO: Task 2052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 508287ed185a66317a85c875024b565a with result from task 2052 parsl.dataflow.dflow DEBUG: Task 2052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/821d87ea-ff40-47a7-9167-d452c56afd49_isrForCrosstalkSources_1031711_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031711/821d87ea-ff40-47a7-9167-d452c56afd49_isrForCrosstalkSources_1031711_48.stderr parsl.dataflow.dflow INFO: Task 2053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f34658522e04e380998148b990c38247 with result from task 2053 parsl.dataflow.dflow DEBUG: Task 2053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/6b3a3f22-dde0-4518-a43e-18ea8dc431c8_isrForCrosstalkSources_1031701_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031701/6b3a3f22-dde0-4518-a43e-18ea8dc431c8_isrForCrosstalkSources_1031701_29.stderr parsl.dataflow.dflow INFO: Task 2046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18b83d566d3568583fdf9fddc95528d4 with result from task 2046 parsl.dataflow.dflow DEBUG: Task 2046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/801eeffb-4ff8-4f0d-99ee-bf6ffa3f4211_isrForCrosstalkSources_1031726_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031726/801eeffb-4ff8-4f0d-99ee-bf6ffa3f4211_isrForCrosstalkSources_1031726_16.stderr parsl.dataflow.dflow INFO: Task 2047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ac71f6d6dfda7494651d1ba14b72428 with result from task 2047 parsl.dataflow.dflow DEBUG: Task 2047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/affe0b7a-5c7a-4bb0-b4e4-e1d1c193dced_isrForCrosstalkSources_1031727_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/affe0b7a-5c7a-4bb0-b4e4-e1d1c193dced_isrForCrosstalkSources_1031727_24.stderr parsl.dataflow.dflow INFO: Task 2044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f369a0c8233273b626247a598239d43d with result from task 2044 parsl.dataflow.dflow DEBUG: Task 2044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cc2a89f7-360f-4402-97e2-7ecb98431a4d_isrForCrosstalkSources_1031727_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031727/cc2a89f7-360f-4402-97e2-7ecb98431a4d_isrForCrosstalkSources_1031727_55.stderr parsl.dataflow.dflow INFO: Task 2042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a210e556bb961c431bb7f376cb2b03fc with result from task 2042 parsl.dataflow.dflow DEBUG: Task 2042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f343e217-a0a6-4b8d-a4ea-6351b4bf7de3_isrForCrosstalkSources_1031705_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031705/f343e217-a0a6-4b8d-a4ea-6351b4bf7de3_isrForCrosstalkSources_1031705_55.stderr parsl.dataflow.dflow INFO: Task 2041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07b4f2a981b426e041e447a3b989cddc with result from task 2041 parsl.dataflow.dflow DEBUG: Task 2041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/9c07404f-0f31-4159-b478-96cacd66af94_isrForCrosstalkSources_1031731_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031731/9c07404f-0f31-4159-b478-96cacd66af94_isrForCrosstalkSources_1031731_42.stderr parsl.dataflow.dflow INFO: Task 2037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec36e0e2cf75b5eb786f6a3535d9f8a5 with result from task 2037 parsl.dataflow.dflow DEBUG: Task 2037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/bd2fa2cf-334b-44d8-8063-1197ac04538b_isrForCrosstalkSources_1031720_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/logs/isrForCrosstalkSources/1031720/bd2fa2cf-334b-44d8-8063-1197ac04538b_isrForCrosstalkSources_1031720_12.stderr parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 2480 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller 2024-03-17 04:07:39 proc_lsst.multi:170 [INFO] [multi] job_id 17148245 proc_lsst.multi INFO: [multi] job_id 17148245 2024-03-17 04:07:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 242 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 242 2024-03-17 04:07:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148245 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148245 parsl.executors.status_handling DEBUG: Launched block 298 on executor multi with job ID 17148245 parsl.executors.status_handling INFO: Allocated block ID 299 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-299 2024-03-17 04:07:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-299 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-299 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 242 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 242 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:39 proc_lsst.multi:170 [INFO] [multi] job_id 17148246 proc_lsst.multi INFO: [multi] job_id 17148246 2024-03-17 04:07:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 243 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 243 2024-03-17 04:07:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148246 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148246 parsl.executors.status_handling DEBUG: Launched block 299 on executor multi with job ID 17148246 parsl.executors.status_handling INFO: Allocated block ID 300 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-300 2024-03-17 04:07:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-300 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-300 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 243 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 243 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:39 proc_lsst.multi:170 [INFO] [multi] job_id 17148247 proc_lsst.multi INFO: [multi] job_id 17148247 2024-03-17 04:07:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 244 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 244 2024-03-17 04:07:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148247 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148247 parsl.executors.status_handling DEBUG: Launched block 300 on executor multi with job ID 17148247 parsl.executors.status_handling INFO: Allocated block ID 301 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-301 2024-03-17 04:07:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-301 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-301 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 244 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 244 2024-03-17 04:07:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:40 proc_lsst.multi:170 [INFO] [multi] job_id 17148248 proc_lsst.multi INFO: [multi] job_id 17148248 2024-03-17 04:07:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 245 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 245 2024-03-17 04:07:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148248 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148248 parsl.executors.status_handling DEBUG: Launched block 301 on executor multi with job ID 17148248 parsl.executors.status_handling INFO: Allocated block ID 302 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-302 2024-03-17 04:07:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-302 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-302 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 245 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 245 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:40 proc_lsst.multi:170 [INFO] [multi] job_id 17148249 proc_lsst.multi INFO: [multi] job_id 17148249 2024-03-17 04:07:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 246 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 246 2024-03-17 04:07:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148249 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148249 parsl.executors.status_handling DEBUG: Launched block 302 on executor multi with job ID 17148249 parsl.executors.status_handling INFO: Allocated block ID 303 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-303 2024-03-17 04:07:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-303 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-303 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 246 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 246 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:40 proc_lsst.multi:170 [INFO] [multi] job_id 17148250 proc_lsst.multi INFO: [multi] job_id 17148250 2024-03-17 04:07:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 247 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 247 2024-03-17 04:07:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148250 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148250 parsl.executors.status_handling DEBUG: Launched block 303 on executor multi with job ID 17148250 parsl.executors.status_handling INFO: Allocated block ID 304 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-304 2024-03-17 04:07:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-304 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-304 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 247 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 247 2024-03-17 04:07:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:41 proc_lsst.multi:170 [INFO] [multi] job_id 17148251 proc_lsst.multi INFO: [multi] job_id 17148251 2024-03-17 04:07:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 248 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 248 2024-03-17 04:07:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148251 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148251 parsl.executors.status_handling DEBUG: Launched block 304 on executor multi with job ID 17148251 parsl.executors.status_handling INFO: Allocated block ID 305 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-305 2024-03-17 04:07:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-305 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-305 2024-03-17 04:07:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 248 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 248 2024-03-17 04:07:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:41 proc_lsst.multi:170 [INFO] [multi] job_id 17148252 proc_lsst.multi INFO: [multi] job_id 17148252 2024-03-17 04:07:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 249 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 249 2024-03-17 04:07:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148252 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148252 parsl.executors.status_handling DEBUG: Launched block 305 on executor multi with job ID 17148252 parsl.executors.status_handling INFO: Allocated block ID 306 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-306 2024-03-17 04:07:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-306 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-306 2024-03-17 04:07:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 249 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 249 2024-03-17 04:07:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:42 proc_lsst.multi:170 [INFO] [multi] job_id 17148253 proc_lsst.multi INFO: [multi] job_id 17148253 2024-03-17 04:07:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 250 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 250 2024-03-17 04:07:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148253 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148253 parsl.executors.status_handling DEBUG: Launched block 306 on executor multi with job ID 17148253 parsl.executors.status_handling INFO: Allocated block ID 307 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-307 2024-03-17 04:07:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-307 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-307 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 250 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 250 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:42 proc_lsst.multi:170 [INFO] [multi] job_id 17148254 proc_lsst.multi INFO: [multi] job_id 17148254 2024-03-17 04:07:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 251 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 251 2024-03-17 04:07:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148254 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148254 parsl.executors.status_handling DEBUG: Launched block 307 on executor multi with job ID 17148254 parsl.executors.status_handling INFO: Allocated block ID 308 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-308 2024-03-17 04:07:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-308 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-308 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 251 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 251 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:42 proc_lsst.multi:170 [INFO] [multi] job_id 17148255 proc_lsst.multi INFO: [multi] job_id 17148255 2024-03-17 04:07:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 252 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 252 2024-03-17 04:07:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148255 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148255 parsl.executors.status_handling DEBUG: Launched block 308 on executor multi with job ID 17148255 parsl.executors.status_handling INFO: Allocated block ID 309 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-309 2024-03-17 04:07:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-309 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-309 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 252 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 252 2024-03-17 04:07:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:43 proc_lsst.multi:170 [INFO] [multi] job_id 17148256 proc_lsst.multi INFO: [multi] job_id 17148256 2024-03-17 04:07:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 253 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 253 2024-03-17 04:07:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148256 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148256 parsl.executors.status_handling DEBUG: Launched block 309 on executor multi with job ID 17148256 parsl.executors.status_handling INFO: Allocated block ID 310 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-310 2024-03-17 04:07:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-310 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-310 2024-03-17 04:07:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 253 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 253 2024-03-17 04:07:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:43 proc_lsst.multi:170 [INFO] [multi] job_id 17148257 proc_lsst.multi INFO: [multi] job_id 17148257 2024-03-17 04:07:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 254 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 254 2024-03-17 04:07:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148257 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148257 parsl.executors.status_handling DEBUG: Launched block 310 on executor multi with job ID 17148257 parsl.executors.status_handling INFO: Allocated block ID 311 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-311 2024-03-17 04:07:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-311 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-311 2024-03-17 04:07:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 04:07:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 04:07:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 04:07:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 04:07:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 04:07:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 04:07:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 04:07:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 04:07:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 04:07:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 04:07:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 254 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 254 2024-03-17 04:07:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,127.0.0.1,n3009,10.64.129.9,10.64.65.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54360 --result_port=54006 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B0c/science#step0/20240317T040120Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 04:07:43 proc_lsst.multi:170 [INFO] [multi] job_id 17148258 proc_lsst.multi INFO: [multi] job_id 17148258 2024-03-17 04:07:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 255 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 255 2024-03-17 04:07:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-17 04:07:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17148258 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17148258 parsl.executors.status_handling DEBUG: Launched block 311 on executor multi with job ID 17148258 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22674350809360 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=311, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 122 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 08c0c54c0ae8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 08c0c54c0ae8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5b9538b04c00 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5b9538b04c00 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df218aa4e3b9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df218aa4e3b9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7d5e5261b91b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7d5e5261b91b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1d2359625da3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1d2359625da3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4496a353e39b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4496a353e39b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a87df60a359b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a87df60a359b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cef7ff5bc788 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cef7ff5bc788 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f9a0ea72693d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f9a0ea72693d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3f078d84a2ed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3f078d84a2ed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f0bdd6f2acb6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f0bdd6f2acb6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: edb83fcbdb57 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: edb83fcbdb57 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6f4b834b0572 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6f4b834b0572 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 86075c0a3003 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 86075c0a3003 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 82f655f2c968 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 82f655f2c968 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cf7561b67038 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cf7561b67038 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dcc403b3d69e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dcc403b3d69e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 354d01e0bb34 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 354d01e0bb34 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9f720b660d3e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9f720b660d3e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f6b948887422 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f6b948887422 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 251b85dc45ef parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 251b85dc45ef parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d2e95af8c52 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d2e95af8c52 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a62a49c3341f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a62a49c3341f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bec49d95b2ee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bec49d95b2ee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 75ad37775ec0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 75ad37775ec0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0f53396abd4c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0f53396abd4c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff638ba01a3f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff638ba01a3f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9b10e67e18b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9b10e67e18b3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b88532bf2e53 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b88532bf2e53 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d3738eb7243a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d3738eb7243a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 77d63cf331d2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 77d63cf331d2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0d400740c71e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0d400740c71e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cbae59d9ea2d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cbae59d9ea2d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 35474ad683ca parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 35474ad683ca parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7d8de646a9bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7d8de646a9bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc70122e1d69 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc70122e1d69 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c53b13a85b1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c53b13a85b1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 799b772bd268 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 799b772bd268 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: afdbfc9648a4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: afdbfc9648a4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 195e80296265 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 195e80296265 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1bf26bb637b6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1bf26bb637b6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bdeba7528ecb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bdeba7528ecb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0621e5266c85 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0621e5266c85 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 121f939df645 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 121f939df645 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 47d4092a0d64 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 47d4092a0d64 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e23b6c956555 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e23b6c956555 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e42862a5cbab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e42862a5cbab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5ccb4ee528a9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5ccb4ee528a9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 74863d9165fb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 74863d9165fb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1a804cceb8e3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1a804cceb8e3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7901d594ead0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7901d594ead0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ede2804bbc42 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ede2804bbc42 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c27fb34a724e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c27fb34a724e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 05712f11dcb4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 05712f11dcb4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ed24c61be07d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ed24c61be07d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 70d537dae37e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 70d537dae37e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 47058a1da24e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 47058a1da24e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce9ddca0f3f5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce9ddca0f3f5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3647b1cefcb6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3647b1cefcb6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 158510af4775 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 158510af4775 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf360bbb0bed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf360bbb0bed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c5c60fa46efe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c5c60fa46efe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dafe1eef4180 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dafe1eef4180 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 32af98a5368d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 32af98a5368d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2f49fad36956 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2f49fad36956 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 36266bec0823 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 36266bec0823 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f9b0a221787d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f9b0a221787d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fa9d50ac5555 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fa9d50ac5555 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d09ceb3173bc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d09ceb3173bc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 783381bd1dc6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 783381bd1dc6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e8c8ab8eed40 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e8c8ab8eed40 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: abba7cb2c1ce parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: abba7cb2c1ce parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5182389b0a92 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5182389b0a92 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9a104f06d6bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9a104f06d6bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a526686b772d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a526686b772d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0ed4919f1af6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0ed4919f1af6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5229f6d6f472 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5229f6d6f472 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d0b4fb751507 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d0b4fb751507 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1a39f98dbe5a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1a39f98dbe5a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 475c589da8e6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 475c589da8e6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 168009fd12c0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 168009fd12c0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 66723a7ab547 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 66723a7ab547 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: db3c9202ccb4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: db3c9202ccb4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d706f87bc055 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d706f87bc055 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c5ef7f04fad2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c5ef7f04fad2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 362dbe64a71a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 362dbe64a71a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df5a1de986b2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df5a1de986b2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 04feebd81990 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 04feebd81990 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b5c86eb7cf2f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b5c86eb7cf2f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 70c55fc46c8c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 70c55fc46c8c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d65559d2caf6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d65559d2caf6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d03b199bfc32 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d03b199bfc32 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ef0f1543be13 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ef0f1543be13 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a9b15b412bf3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a9b15b412bf3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1993e4bac0c6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1993e4bac0c6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8413914cac17 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8413914cac17 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a1a27ab266aa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a1a27ab266aa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 68ef1cbd3b8d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 68ef1cbd3b8d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 04baa55b8f1f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 04baa55b8f1f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dad5ee1581a5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dad5ee1581a5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a0ca25c72134 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a0ca25c72134 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2e4339a2a11f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2e4339a2a11f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 10ea907cdb3e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 10ea907cdb3e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b14eba38a37a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b14eba38a37a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cec362f09015 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cec362f09015 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4bc5179f200b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4bc5179f200b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b7cd57928d46 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b7cd57928d46 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 741a647da174 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 741a647da174 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aabda76dc755 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aabda76dc755 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8909ddebfe45 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8909ddebfe45 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0415045b15d3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0415045b15d3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 412a1c8f4132 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 412a1c8f4132 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9eb0dd79abcc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9eb0dd79abcc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3395111e550e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3395111e550e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1f1f864213ef parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1f1f864213ef parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8f37b399bc7b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8f37b399bc7b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9631affd3429 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9631affd3429 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 967b36c8e598 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 967b36c8e598 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 060552d0715d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 060552d0715d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d369a2546e22 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d369a2546e22 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e568ce1ea3fa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e568ce1ea3fa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 69da47f30ec6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 69da47f30ec6 2024-03-17 04:07:43 proc_lsst.multi:146 [INFO] found job 17148066 in provider checkpoint proc_lsst.multi INFO: found job 17148066 in provider checkpoint 2024-03-17 04:07:43 proc_lsst.multi:201 [INFO] cancelling 17148066 on provider checkpoint proc_lsst.multi INFO: cancelling 17148066 on provider checkpoint 2024-03-17 04:07:44 proc_lsst.multi:146 [INFO] found job 17148065 in provider checkpoint proc_lsst.multi INFO: found job 17148065 in provider checkpoint 2024-03-17 04:07:44 proc_lsst.multi:201 [INFO] cancelling 17148065 on provider checkpoint proc_lsst.multi INFO: cancelling 17148065 on provider checkpoint 2024-03-17 04:07:44 proc_lsst.multi:146 [INFO] found job 17148067 in provider checkpoint proc_lsst.multi INFO: found job 17148067 in provider checkpoint 2024-03-17 04:07:44 proc_lsst.multi:201 [INFO] cancelling 17148067 on provider checkpoint proc_lsst.multi INFO: cancelling 17148067 on provider checkpoint 2024-03-17 04:07:44 proc_lsst.multi:146 [INFO] found job 17148015 in provider checkpoint proc_lsst.multi INFO: found job 17148015 in provider checkpoint 2024-03-17 04:07:44 proc_lsst.multi:201 [INFO] cancelling 17148015 on provider checkpoint proc_lsst.multi INFO: cancelling 17148015 on provider checkpoint 2024-03-17 04:07:45 proc_lsst.multi:146 [INFO] found job 17148013 in provider checkpoint proc_lsst.multi INFO: found job 17148013 in provider checkpoint 2024-03-17 04:07:45 proc_lsst.multi:201 [INFO] cancelling 17148013 on provider checkpoint proc_lsst.multi INFO: cancelling 17148013 on provider checkpoint 2024-03-17 04:07:45 proc_lsst.multi:146 [INFO] found job 17148014 in provider checkpoint proc_lsst.multi INFO: found job 17148014 in provider checkpoint 2024-03-17 04:07:45 proc_lsst.multi:201 [INFO] cancelling 17148014 on provider checkpoint proc_lsst.multi INFO: cancelling 17148014 on provider checkpoint 2024-03-17 04:07:45 proc_lsst.multi:146 [INFO] found job 17148005 in provider checkpoint proc_lsst.multi INFO: found job 17148005 in provider checkpoint 2024-03-17 04:07:45 proc_lsst.multi:201 [INFO] cancelling 17148005 on provider checkpoint proc_lsst.multi INFO: cancelling 17148005 on provider checkpoint 2024-03-17 04:07:46 proc_lsst.multi:146 [INFO] found job 17148016 in provider checkpoint proc_lsst.multi INFO: found job 17148016 in provider checkpoint 2024-03-17 04:07:46 proc_lsst.multi:201 [INFO] cancelling 17148016 on provider checkpoint proc_lsst.multi INFO: cancelling 17148016 on provider checkpoint 2024-03-17 04:07:46 proc_lsst.multi:146 [INFO] found job 17148017 in provider checkpoint proc_lsst.multi INFO: found job 17148017 in provider checkpoint 2024-03-17 04:07:46 proc_lsst.multi:201 [INFO] cancelling 17148017 on provider checkpoint proc_lsst.multi INFO: cancelling 17148017 on provider checkpoint 2024-03-17 04:07:46 proc_lsst.multi:146 [INFO] found job 17148018 in provider checkpoint proc_lsst.multi INFO: found job 17148018 in provider checkpoint 2024-03-17 04:07:46 proc_lsst.multi:201 [INFO] cancelling 17148018 on provider checkpoint proc_lsst.multi INFO: cancelling 17148018 on provider checkpoint 2024-03-17 04:07:47 proc_lsst.multi:146 [INFO] found job 17148023 in provider checkpoint proc_lsst.multi INFO: found job 17148023 in provider checkpoint 2024-03-17 04:07:47 proc_lsst.multi:201 [INFO] cancelling 17148023 on provider checkpoint proc_lsst.multi INFO: cancelling 17148023 on provider checkpoint 2024-03-17 04:07:47 proc_lsst.multi:146 [INFO] found job 17148024 in provider checkpoint proc_lsst.multi INFO: found job 17148024 in provider checkpoint 2024-03-17 04:07:47 proc_lsst.multi:201 [INFO] cancelling 17148024 on provider checkpoint proc_lsst.multi INFO: cancelling 17148024 on provider checkpoint 2024-03-17 04:07:47 proc_lsst.multi:146 [INFO] found job 17147933 in provider astro proc_lsst.multi INFO: found job 17147933 in provider astro 2024-03-17 04:07:47 proc_lsst.multi:201 [INFO] cancelling 17147933 on provider astro proc_lsst.multi INFO: cancelling 17147933 on provider astro 2024-03-17 04:07:48 proc_lsst.multi:146 [INFO] found job 17147959 in provider astro proc_lsst.multi INFO: found job 17147959 in provider astro 2024-03-17 04:07:48 proc_lsst.multi:201 [INFO] cancelling 17147959 on provider astro proc_lsst.multi INFO: cancelling 17147959 on provider astro 2024-03-17 04:07:48 proc_lsst.multi:146 [INFO] found job 17147956 in provider astro proc_lsst.multi INFO: found job 17147956 in provider astro 2024-03-17 04:07:48 proc_lsst.multi:201 [INFO] cancelling 17147956 on provider astro proc_lsst.multi INFO: cancelling 17147956 on provider astro 2024-03-17 04:07:49 proc_lsst.multi:146 [INFO] found job 17148026 in provider checkpoint proc_lsst.multi INFO: found job 17148026 in provider checkpoint 2024-03-17 04:07:49 proc_lsst.multi:201 [INFO] cancelling 17148026 on provider checkpoint proc_lsst.multi INFO: cancelling 17148026 on provider checkpoint 2024-03-17 04:07:49 proc_lsst.multi:146 [INFO] found job 17147942 in provider astro proc_lsst.multi INFO: found job 17147942 in provider astro 2024-03-17 04:07:49 proc_lsst.multi:201 [INFO] cancelling 17147942 on provider astro proc_lsst.multi INFO: cancelling 17147942 on provider astro 2024-03-17 04:07:49 proc_lsst.multi:146 [INFO] found job 17148025 in provider checkpoint proc_lsst.multi INFO: found job 17148025 in provider checkpoint 2024-03-17 04:07:49 proc_lsst.multi:201 [INFO] cancelling 17148025 on provider checkpoint proc_lsst.multi INFO: cancelling 17148025 on provider checkpoint 2024-03-17 04:07:50 proc_lsst.multi:146 [INFO] found job 17147953 in provider astro proc_lsst.multi INFO: found job 17147953 in provider astro 2024-03-17 04:07:50 proc_lsst.multi:201 [INFO] cancelling 17147953 on provider astro proc_lsst.multi INFO: cancelling 17147953 on provider astro 2024-03-17 04:07:50 proc_lsst.multi:146 [INFO] found job 17147950 in provider astro proc_lsst.multi INFO: found job 17147950 in provider astro 2024-03-17 04:07:50 proc_lsst.multi:201 [INFO] cancelling 17147950 on provider astro proc_lsst.multi INFO: cancelling 17147950 on provider astro 2024-03-17 04:07:50 proc_lsst.multi:146 [INFO] found job 17147932 in provider astro proc_lsst.multi INFO: found job 17147932 in provider astro 2024-03-17 04:07:50 proc_lsst.multi:201 [INFO] cancelling 17147932 on provider astro proc_lsst.multi INFO: cancelling 17147932 on provider astro 2024-03-17 04:07:51 proc_lsst.multi:146 [INFO] found job 17148042 in provider checkpoint proc_lsst.multi INFO: found job 17148042 in provider checkpoint 2024-03-17 04:07:51 proc_lsst.multi:201 [INFO] cancelling 17148042 on provider checkpoint proc_lsst.multi INFO: cancelling 17148042 on provider checkpoint 2024-03-17 04:07:51 proc_lsst.multi:146 [INFO] found job 17147940 in provider astro proc_lsst.multi INFO: found job 17147940 in provider astro 2024-03-17 04:07:51 proc_lsst.multi:201 [INFO] cancelling 17147940 on provider astro proc_lsst.multi INFO: cancelling 17147940 on provider astro 2024-03-17 04:07:51 proc_lsst.multi:146 [INFO] found job 17147943 in provider astro proc_lsst.multi INFO: found job 17147943 in provider astro 2024-03-17 04:07:51 proc_lsst.multi:201 [INFO] cancelling 17147943 on provider astro proc_lsst.multi INFO: cancelling 17147943 on provider astro 2024-03-17 04:07:52 proc_lsst.multi:146 [INFO] found job 17147934 in provider astro proc_lsst.multi INFO: found job 17147934 in provider astro 2024-03-17 04:07:52 proc_lsst.multi:201 [INFO] cancelling 17147934 on provider astro proc_lsst.multi INFO: cancelling 17147934 on provider astro 2024-03-17 04:07:52 proc_lsst.multi:146 [INFO] found job 17148041 in provider checkpoint proc_lsst.multi INFO: found job 17148041 in provider checkpoint 2024-03-17 04:07:52 proc_lsst.multi:201 [INFO] cancelling 17148041 on provider checkpoint proc_lsst.multi INFO: cancelling 17148041 on provider checkpoint 2024-03-17 04:07:52 proc_lsst.multi:146 [INFO] found job 17147947 in provider astro proc_lsst.multi INFO: found job 17147947 in provider astro 2024-03-17 04:07:52 proc_lsst.multi:201 [INFO] cancelling 17147947 on provider astro proc_lsst.multi INFO: cancelling 17147947 on provider astro 2024-03-17 04:07:52 proc_lsst.multi:146 [INFO] found job 17148029 in provider checkpoint proc_lsst.multi INFO: found job 17148029 in provider checkpoint 2024-03-17 04:07:52 proc_lsst.multi:201 [INFO] cancelling 17148029 on provider checkpoint proc_lsst.multi INFO: cancelling 17148029 on provider checkpoint 2024-03-17 04:07:53 proc_lsst.multi:146 [INFO] found job 17147945 in provider astro proc_lsst.multi INFO: found job 17147945 in provider astro 2024-03-17 04:07:53 proc_lsst.multi:201 [INFO] cancelling 17147945 on provider astro proc_lsst.multi INFO: cancelling 17147945 on provider astro 2024-03-17 04:07:53 proc_lsst.multi:146 [INFO] found job 17147937 in provider astro proc_lsst.multi INFO: found job 17147937 in provider astro 2024-03-17 04:07:53 proc_lsst.multi:201 [INFO] cancelling 17147937 on provider astro proc_lsst.multi INFO: cancelling 17147937 on provider astro 2024-03-17 04:07:53 proc_lsst.multi:146 [INFO] found job 17148055 in provider checkpoint proc_lsst.multi INFO: found job 17148055 in provider checkpoint 2024-03-17 04:07:53 proc_lsst.multi:201 [INFO] cancelling 17148055 on provider checkpoint proc_lsst.multi INFO: cancelling 17148055 on provider checkpoint 2024-03-17 04:07:54 proc_lsst.multi:146 [INFO] found job 17148054 in provider checkpoint proc_lsst.multi INFO: found job 17148054 in provider checkpoint 2024-03-17 04:07:54 proc_lsst.multi:201 [INFO] cancelling 17148054 on provider checkpoint proc_lsst.multi INFO: cancelling 17148054 on provider checkpoint 2024-03-17 04:07:54 proc_lsst.multi:146 [INFO] found job 17148057 in provider checkpoint proc_lsst.multi INFO: found job 17148057 in provider checkpoint 2024-03-17 04:07:54 proc_lsst.multi:201 [INFO] cancelling 17148057 on provider checkpoint proc_lsst.multi INFO: cancelling 17148057 on provider checkpoint 2024-03-17 04:07:54 proc_lsst.multi:146 [INFO] found job 17148064 in provider checkpoint proc_lsst.multi INFO: found job 17148064 in provider checkpoint 2024-03-17 04:07:54 proc_lsst.multi:201 [INFO] cancelling 17148064 on provider checkpoint proc_lsst.multi INFO: cancelling 17148064 on provider checkpoint 2024-03-17 04:07:54 proc_lsst.multi:146 [INFO] found job 17148053 in provider checkpoint proc_lsst.multi INFO: found job 17148053 in provider checkpoint 2024-03-17 04:07:54 proc_lsst.multi:201 [INFO] cancelling 17148053 on provider checkpoint proc_lsst.multi INFO: cancelling 17148053 on provider checkpoint 2024-03-17 04:07:55 proc_lsst.multi:146 [INFO] found job 17147951 in provider astro proc_lsst.multi INFO: found job 17147951 in provider astro 2024-03-17 04:07:55 proc_lsst.multi:201 [INFO] cancelling 17147951 on provider astro proc_lsst.multi INFO: cancelling 17147951 on provider astro 2024-03-17 04:07:55 proc_lsst.multi:146 [INFO] found job 17147989 in provider escience_a40 proc_lsst.multi INFO: found job 17147989 in provider escience_a40 2024-03-17 04:07:55 proc_lsst.multi:201 [INFO] cancelling 17147989 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147989 on provider escience_a40 2024-03-17 04:07:55 proc_lsst.multi:146 [INFO] found job 17148049 in provider checkpoint proc_lsst.multi INFO: found job 17148049 in provider checkpoint 2024-03-17 04:07:55 proc_lsst.multi:201 [INFO] cancelling 17148049 on provider checkpoint proc_lsst.multi INFO: cancelling 17148049 on provider checkpoint 2024-03-17 04:07:55 proc_lsst.multi:146 [INFO] found job 17147938 in provider astro proc_lsst.multi INFO: found job 17147938 in provider astro 2024-03-17 04:07:55 proc_lsst.multi:201 [INFO] cancelling 17147938 on provider astro proc_lsst.multi INFO: cancelling 17147938 on provider astro 2024-03-17 04:07:56 proc_lsst.multi:146 [INFO] found job 17148030 in provider checkpoint proc_lsst.multi INFO: found job 17148030 in provider checkpoint 2024-03-17 04:07:56 proc_lsst.multi:201 [INFO] cancelling 17148030 on provider checkpoint proc_lsst.multi INFO: cancelling 17148030 on provider checkpoint 2024-03-17 04:07:56 proc_lsst.multi:146 [INFO] found job 3544 in provider local proc_lsst.multi INFO: found job 3544 in provider local 2024-03-17 04:07:56 proc_lsst.multi:201 [INFO] cancelling 3544 on provider local proc_lsst.multi INFO: cancelling 3544 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 3544 2024-03-17 04:07:56 proc_lsst.multi:146 [INFO] found job 17148050 in provider checkpoint proc_lsst.multi INFO: found job 17148050 in provider checkpoint 2024-03-17 04:07:56 proc_lsst.multi:201 [INFO] cancelling 17148050 on provider checkpoint proc_lsst.multi INFO: cancelling 17148050 on provider checkpoint 2024-03-17 04:07:56 proc_lsst.multi:146 [INFO] found job 17147955 in provider astro proc_lsst.multi INFO: found job 17147955 in provider astro 2024-03-17 04:07:56 proc_lsst.multi:201 [INFO] cancelling 17147955 on provider astro proc_lsst.multi INFO: cancelling 17147955 on provider astro 2024-03-17 04:07:57 proc_lsst.multi:146 [INFO] found job 17148062 in provider checkpoint proc_lsst.multi INFO: found job 17148062 in provider checkpoint 2024-03-17 04:07:57 proc_lsst.multi:201 [INFO] cancelling 17148062 on provider checkpoint proc_lsst.multi INFO: cancelling 17148062 on provider checkpoint 2024-03-17 04:07:57 proc_lsst.multi:146 [INFO] found job 17147935 in provider astro proc_lsst.multi INFO: found job 17147935 in provider astro 2024-03-17 04:07:57 proc_lsst.multi:201 [INFO] cancelling 17147935 on provider astro proc_lsst.multi INFO: cancelling 17147935 on provider astro 2024-03-17 04:07:57 proc_lsst.multi:146 [INFO] found job 17147984 in provider escience_a40 proc_lsst.multi INFO: found job 17147984 in provider escience_a40 2024-03-17 04:07:57 proc_lsst.multi:201 [INFO] cancelling 17147984 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147984 on provider escience_a40 2024-03-17 04:07:58 proc_lsst.multi:146 [INFO] found job 17147992 in provider escience_a40 proc_lsst.multi INFO: found job 17147992 in provider escience_a40 2024-03-17 04:07:58 proc_lsst.multi:201 [INFO] cancelling 17147992 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147992 on provider escience_a40 2024-03-17 04:07:58 proc_lsst.multi:146 [INFO] found job 17148004 in provider checkpoint proc_lsst.multi INFO: found job 17148004 in provider checkpoint 2024-03-17 04:07:58 proc_lsst.multi:201 [INFO] cancelling 17148004 on provider checkpoint proc_lsst.multi INFO: cancelling 17148004 on provider checkpoint 2024-03-17 04:07:58 proc_lsst.multi:146 [INFO] found job 17148063 in provider checkpoint proc_lsst.multi INFO: found job 17148063 in provider checkpoint 2024-03-17 04:07:58 proc_lsst.multi:201 [INFO] cancelling 17148063 on provider checkpoint proc_lsst.multi INFO: cancelling 17148063 on provider checkpoint 2024-03-17 04:07:59 proc_lsst.multi:146 [INFO] found job 17147996 in provider escience_a40 proc_lsst.multi INFO: found job 17147996 in provider escience_a40 2024-03-17 04:07:59 proc_lsst.multi:201 [INFO] cancelling 17147996 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147996 on provider escience_a40 2024-03-17 04:07:59 proc_lsst.multi:146 [INFO] found job 17147954 in provider astro proc_lsst.multi INFO: found job 17147954 in provider astro 2024-03-17 04:07:59 proc_lsst.multi:201 [INFO] cancelling 17147954 on provider astro proc_lsst.multi INFO: cancelling 17147954 on provider astro 2024-03-17 04:07:59 proc_lsst.multi:146 [INFO] found job 17147931 in provider astro proc_lsst.multi INFO: found job 17147931 in provider astro 2024-03-17 04:07:59 proc_lsst.multi:201 [INFO] cancelling 17147931 on provider astro proc_lsst.multi INFO: cancelling 17147931 on provider astro 2024-03-17 04:08:00 proc_lsst.multi:146 [INFO] found job 17148031 in provider checkpoint proc_lsst.multi INFO: found job 17148031 in provider checkpoint 2024-03-17 04:08:00 proc_lsst.multi:201 [INFO] cancelling 17148031 on provider checkpoint proc_lsst.multi INFO: cancelling 17148031 on provider checkpoint 2024-03-17 04:08:00 proc_lsst.multi:146 [INFO] found job 17148032 in provider checkpoint proc_lsst.multi INFO: found job 17148032 in provider checkpoint 2024-03-17 04:08:00 proc_lsst.multi:201 [INFO] cancelling 17148032 on provider checkpoint proc_lsst.multi INFO: cancelling 17148032 on provider checkpoint 2024-03-17 04:08:01 proc_lsst.multi:146 [INFO] found job 17147946 in provider astro proc_lsst.multi INFO: found job 17147946 in provider astro 2024-03-17 04:08:01 proc_lsst.multi:201 [INFO] cancelling 17147946 on provider astro proc_lsst.multi INFO: cancelling 17147946 on provider astro 2024-03-17 04:08:01 proc_lsst.multi:146 [INFO] found job 17147977 in provider escience_a40 proc_lsst.multi INFO: found job 17147977 in provider escience_a40 2024-03-17 04:08:01 proc_lsst.multi:201 [INFO] cancelling 17147977 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147977 on provider escience_a40 2024-03-17 04:08:01 proc_lsst.multi:146 [INFO] found job 17148056 in provider checkpoint proc_lsst.multi INFO: found job 17148056 in provider checkpoint 2024-03-17 04:08:01 proc_lsst.multi:201 [INFO] cancelling 17148056 on provider checkpoint proc_lsst.multi INFO: cancelling 17148056 on provider checkpoint 2024-03-17 04:08:02 proc_lsst.multi:146 [INFO] found job 17147944 in provider astro proc_lsst.multi INFO: found job 17147944 in provider astro 2024-03-17 04:08:02 proc_lsst.multi:201 [INFO] cancelling 17147944 on provider astro proc_lsst.multi INFO: cancelling 17147944 on provider astro 2024-03-17 04:08:02 proc_lsst.multi:146 [INFO] found job 17147988 in provider escience_a40 proc_lsst.multi INFO: found job 17147988 in provider escience_a40 2024-03-17 04:08:02 proc_lsst.multi:201 [INFO] cancelling 17147988 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147988 on provider escience_a40 2024-03-17 04:08:02 proc_lsst.multi:146 [INFO] found job 17147985 in provider escience_a40 proc_lsst.multi INFO: found job 17147985 in provider escience_a40 2024-03-17 04:08:02 proc_lsst.multi:201 [INFO] cancelling 17147985 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147985 on provider escience_a40 2024-03-17 04:08:03 proc_lsst.multi:146 [INFO] found job 17148036 in provider checkpoint proc_lsst.multi INFO: found job 17148036 in provider checkpoint 2024-03-17 04:08:03 proc_lsst.multi:201 [INFO] cancelling 17148036 on provider checkpoint proc_lsst.multi INFO: cancelling 17148036 on provider checkpoint 2024-03-17 04:08:03 proc_lsst.multi:146 [INFO] found job 17147958 in provider astro proc_lsst.multi INFO: found job 17147958 in provider astro 2024-03-17 04:08:03 proc_lsst.multi:201 [INFO] cancelling 17147958 on provider astro proc_lsst.multi INFO: cancelling 17147958 on provider astro 2024-03-17 04:08:03 proc_lsst.multi:146 [INFO] found job 17148035 in provider checkpoint proc_lsst.multi INFO: found job 17148035 in provider checkpoint 2024-03-17 04:08:03 proc_lsst.multi:201 [INFO] cancelling 17148035 on provider checkpoint proc_lsst.multi INFO: cancelling 17148035 on provider checkpoint 2024-03-17 04:08:03 proc_lsst.multi:146 [INFO] found job 17148046 in provider checkpoint proc_lsst.multi INFO: found job 17148046 in provider checkpoint 2024-03-17 04:08:03 proc_lsst.multi:201 [INFO] cancelling 17148046 on provider checkpoint proc_lsst.multi INFO: cancelling 17148046 on provider checkpoint 2024-03-17 04:08:04 proc_lsst.multi:146 [INFO] found job 17148045 in provider checkpoint proc_lsst.multi INFO: found job 17148045 in provider checkpoint 2024-03-17 04:08:04 proc_lsst.multi:201 [INFO] cancelling 17148045 on provider checkpoint proc_lsst.multi INFO: cancelling 17148045 on provider checkpoint 2024-03-17 04:08:04 proc_lsst.multi:146 [INFO] found job 17148059 in provider checkpoint proc_lsst.multi INFO: found job 17148059 in provider checkpoint 2024-03-17 04:08:04 proc_lsst.multi:201 [INFO] cancelling 17148059 on provider checkpoint proc_lsst.multi INFO: cancelling 17148059 on provider checkpoint 2024-03-17 04:08:04 proc_lsst.multi:146 [INFO] found job 17148058 in provider checkpoint proc_lsst.multi INFO: found job 17148058 in provider checkpoint 2024-03-17 04:08:04 proc_lsst.multi:201 [INFO] cancelling 17148058 on provider checkpoint proc_lsst.multi INFO: cancelling 17148058 on provider checkpoint 2024-03-17 04:08:05 proc_lsst.multi:146 [INFO] found job 17148044 in provider checkpoint proc_lsst.multi INFO: found job 17148044 in provider checkpoint 2024-03-17 04:08:05 proc_lsst.multi:201 [INFO] cancelling 17148044 on provider checkpoint proc_lsst.multi INFO: cancelling 17148044 on provider checkpoint 2024-03-17 04:08:05 proc_lsst.multi:146 [INFO] found job 17147948 in provider astro proc_lsst.multi INFO: found job 17147948 in provider astro 2024-03-17 04:08:05 proc_lsst.multi:201 [INFO] cancelling 17147948 on provider astro proc_lsst.multi INFO: cancelling 17147948 on provider astro 2024-03-17 04:08:05 proc_lsst.multi:146 [INFO] found job 17148043 in provider checkpoint proc_lsst.multi INFO: found job 17148043 in provider checkpoint 2024-03-17 04:08:05 proc_lsst.multi:201 [INFO] cancelling 17148043 on provider checkpoint proc_lsst.multi INFO: cancelling 17148043 on provider checkpoint 2024-03-17 04:08:06 proc_lsst.multi:146 [INFO] found job 17147957 in provider astro proc_lsst.multi INFO: found job 17147957 in provider astro 2024-03-17 04:08:06 proc_lsst.multi:201 [INFO] cancelling 17147957 on provider astro proc_lsst.multi INFO: cancelling 17147957 on provider astro 2024-03-17 04:08:06 proc_lsst.multi:146 [INFO] found job 17148006 in provider checkpoint proc_lsst.multi INFO: found job 17148006 in provider checkpoint 2024-03-17 04:08:06 proc_lsst.multi:201 [INFO] cancelling 17148006 on provider checkpoint proc_lsst.multi INFO: cancelling 17148006 on provider checkpoint 2024-03-17 04:08:06 proc_lsst.multi:146 [INFO] found job 17147941 in provider astro proc_lsst.multi INFO: found job 17147941 in provider astro 2024-03-17 04:08:06 proc_lsst.multi:201 [INFO] cancelling 17147941 on provider astro proc_lsst.multi INFO: cancelling 17147941 on provider astro 2024-03-17 04:08:06 proc_lsst.multi:146 [INFO] found job 17147995 in provider escience_a40 proc_lsst.multi INFO: found job 17147995 in provider escience_a40 2024-03-17 04:08:06 proc_lsst.multi:201 [INFO] cancelling 17147995 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147995 on provider escience_a40 2024-03-17 04:08:07 proc_lsst.multi:146 [INFO] found job 17147993 in provider escience_a40 proc_lsst.multi INFO: found job 17147993 in provider escience_a40 2024-03-17 04:08:07 proc_lsst.multi:201 [INFO] cancelling 17147993 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147993 on provider escience_a40 2024-03-17 04:08:07 proc_lsst.multi:146 [INFO] found job 17148040 in provider checkpoint proc_lsst.multi INFO: found job 17148040 in provider checkpoint 2024-03-17 04:08:07 proc_lsst.multi:201 [INFO] cancelling 17148040 on provider checkpoint proc_lsst.multi INFO: cancelling 17148040 on provider checkpoint 2024-03-17 04:08:07 proc_lsst.multi:146 [INFO] found job 17148012 in provider checkpoint proc_lsst.multi INFO: found job 17148012 in provider checkpoint 2024-03-17 04:08:07 proc_lsst.multi:201 [INFO] cancelling 17148012 on provider checkpoint proc_lsst.multi INFO: cancelling 17148012 on provider checkpoint 2024-03-17 04:08:08 proc_lsst.multi:146 [INFO] found job 17148009 in provider checkpoint proc_lsst.multi INFO: found job 17148009 in provider checkpoint 2024-03-17 04:08:08 proc_lsst.multi:201 [INFO] cancelling 17148009 on provider checkpoint proc_lsst.multi INFO: cancelling 17148009 on provider checkpoint 2024-03-17 04:08:08 proc_lsst.multi:146 [INFO] found job 17147997 in provider escience_a40 proc_lsst.multi INFO: found job 17147997 in provider escience_a40 2024-03-17 04:08:08 proc_lsst.multi:201 [INFO] cancelling 17147997 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147997 on provider escience_a40 2024-03-17 04:08:08 proc_lsst.multi:146 [INFO] found job 17148008 in provider checkpoint proc_lsst.multi INFO: found job 17148008 in provider checkpoint 2024-03-17 04:08:08 proc_lsst.multi:201 [INFO] cancelling 17148008 on provider checkpoint proc_lsst.multi INFO: cancelling 17148008 on provider checkpoint 2024-03-17 04:08:09 proc_lsst.multi:146 [INFO] found job 17148011 in provider checkpoint proc_lsst.multi INFO: found job 17148011 in provider checkpoint 2024-03-17 04:08:09 proc_lsst.multi:201 [INFO] cancelling 17148011 on provider checkpoint proc_lsst.multi INFO: cancelling 17148011 on provider checkpoint 2024-03-17 04:08:09 proc_lsst.multi:146 [INFO] found job 17148003 in provider checkpoint proc_lsst.multi INFO: found job 17148003 in provider checkpoint 2024-03-17 04:08:09 proc_lsst.multi:201 [INFO] cancelling 17148003 on provider checkpoint proc_lsst.multi INFO: cancelling 17148003 on provider checkpoint 2024-03-17 04:08:09 proc_lsst.multi:146 [INFO] found job 17148039 in provider checkpoint proc_lsst.multi INFO: found job 17148039 in provider checkpoint 2024-03-17 04:08:09 proc_lsst.multi:201 [INFO] cancelling 17148039 on provider checkpoint proc_lsst.multi INFO: cancelling 17148039 on provider checkpoint 2024-03-17 04:08:10 proc_lsst.multi:146 [INFO] found job 17148061 in provider checkpoint proc_lsst.multi INFO: found job 17148061 in provider checkpoint 2024-03-17 04:08:10 proc_lsst.multi:201 [INFO] cancelling 17148061 on provider checkpoint proc_lsst.multi INFO: cancelling 17148061 on provider checkpoint 2024-03-17 04:08:10 proc_lsst.multi:146 [INFO] found job 17147952 in provider astro proc_lsst.multi INFO: found job 17147952 in provider astro 2024-03-17 04:08:10 proc_lsst.multi:201 [INFO] cancelling 17147952 on provider astro proc_lsst.multi INFO: cancelling 17147952 on provider astro 2024-03-17 04:08:10 proc_lsst.multi:146 [INFO] found job 17148010 in provider checkpoint proc_lsst.multi INFO: found job 17148010 in provider checkpoint 2024-03-17 04:08:10 proc_lsst.multi:201 [INFO] cancelling 17148010 on provider checkpoint proc_lsst.multi INFO: cancelling 17148010 on provider checkpoint 2024-03-17 04:08:10 proc_lsst.multi:146 [INFO] found job 17147936 in provider astro proc_lsst.multi INFO: found job 17147936 in provider astro 2024-03-17 04:08:10 proc_lsst.multi:201 [INFO] cancelling 17147936 on provider astro proc_lsst.multi INFO: cancelling 17147936 on provider astro 2024-03-17 04:08:11 proc_lsst.multi:146 [INFO] found job 17148007 in provider checkpoint proc_lsst.multi INFO: found job 17148007 in provider checkpoint 2024-03-17 04:08:11 proc_lsst.multi:201 [INFO] cancelling 17148007 on provider checkpoint proc_lsst.multi INFO: cancelling 17148007 on provider checkpoint 2024-03-17 04:08:11 proc_lsst.multi:146 [INFO] found job 17148000 in provider escience_a40 proc_lsst.multi INFO: found job 17148000 in provider escience_a40 2024-03-17 04:08:11 proc_lsst.multi:201 [INFO] cancelling 17148000 on provider escience_a40 proc_lsst.multi INFO: cancelling 17148000 on provider escience_a40 2024-03-17 04:08:11 proc_lsst.multi:146 [INFO] found job 17148021 in provider checkpoint proc_lsst.multi INFO: found job 17148021 in provider checkpoint 2024-03-17 04:08:11 proc_lsst.multi:201 [INFO] cancelling 17148021 on provider checkpoint proc_lsst.multi INFO: cancelling 17148021 on provider checkpoint 2024-03-17 04:08:12 proc_lsst.multi:146 [INFO] found job 17148033 in provider checkpoint proc_lsst.multi INFO: found job 17148033 in provider checkpoint 2024-03-17 04:08:12 proc_lsst.multi:201 [INFO] cancelling 17148033 on provider checkpoint proc_lsst.multi INFO: cancelling 17148033 on provider checkpoint 2024-03-17 04:08:12 proc_lsst.multi:146 [INFO] found job 17148037 in provider checkpoint proc_lsst.multi INFO: found job 17148037 in provider checkpoint 2024-03-17 04:08:12 proc_lsst.multi:201 [INFO] cancelling 17148037 on provider checkpoint proc_lsst.multi INFO: cancelling 17148037 on provider checkpoint 2024-03-17 04:08:12 proc_lsst.multi:146 [INFO] found job 17147980 in provider escience_a40 proc_lsst.multi INFO: found job 17147980 in provider escience_a40 2024-03-17 04:08:12 proc_lsst.multi:201 [INFO] cancelling 17147980 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147980 on provider escience_a40 2024-03-17 04:08:13 proc_lsst.multi:146 [INFO] found job 17148027 in provider checkpoint proc_lsst.multi INFO: found job 17148027 in provider checkpoint 2024-03-17 04:08:13 proc_lsst.multi:201 [INFO] cancelling 17148027 on provider checkpoint proc_lsst.multi INFO: cancelling 17148027 on provider checkpoint 2024-03-17 04:08:13 proc_lsst.multi:146 [INFO] found job 17147978 in provider escience_a40 proc_lsst.multi INFO: found job 17147978 in provider escience_a40 2024-03-17 04:08:13 proc_lsst.multi:201 [INFO] cancelling 17147978 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147978 on provider escience_a40 2024-03-17 04:08:13 proc_lsst.multi:146 [INFO] found job 17148038 in provider checkpoint proc_lsst.multi INFO: found job 17148038 in provider checkpoint 2024-03-17 04:08:13 proc_lsst.multi:201 [INFO] cancelling 17148038 on provider checkpoint proc_lsst.multi INFO: cancelling 17148038 on provider checkpoint 2024-03-17 04:08:14 proc_lsst.multi:146 [INFO] found job 17147986 in provider escience_a40 proc_lsst.multi INFO: found job 17147986 in provider escience_a40 2024-03-17 04:08:14 proc_lsst.multi:201 [INFO] cancelling 17147986 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147986 on provider escience_a40 2024-03-17 04:08:14 proc_lsst.multi:146 [INFO] found job 17147994 in provider escience_a40 proc_lsst.multi INFO: found job 17147994 in provider escience_a40 2024-03-17 04:08:14 proc_lsst.multi:201 [INFO] cancelling 17147994 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147994 on provider escience_a40 2024-03-17 04:08:14 proc_lsst.multi:146 [INFO] found job 17147963 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147963 in provider escience_rtx6k 2024-03-17 04:08:14 proc_lsst.multi:201 [INFO] cancelling 17147963 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17147963 on provider escience_rtx6k 2024-03-17 04:08:14 proc_lsst.multi:146 [INFO] found job 17147965 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147965 in provider escience_rtx6k 2024-03-17 04:08:14 proc_lsst.multi:201 [INFO] cancelling 17147965 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17147965 on provider escience_rtx6k 2024-03-17 04:08:15 proc_lsst.multi:146 [INFO] found job 17148051 in provider checkpoint proc_lsst.multi INFO: found job 17148051 in provider checkpoint 2024-03-17 04:08:15 proc_lsst.multi:201 [INFO] cancelling 17148051 on provider checkpoint proc_lsst.multi INFO: cancelling 17148051 on provider checkpoint 2024-03-17 04:08:15 proc_lsst.multi:146 [INFO] found job 17148028 in provider checkpoint proc_lsst.multi INFO: found job 17148028 in provider checkpoint 2024-03-17 04:08:15 proc_lsst.multi:201 [INFO] cancelling 17148028 on provider checkpoint proc_lsst.multi INFO: cancelling 17148028 on provider checkpoint 2024-03-17 04:08:15 proc_lsst.multi:146 [INFO] found job 17147981 in provider escience_a40 proc_lsst.multi INFO: found job 17147981 in provider escience_a40 2024-03-17 04:08:15 proc_lsst.multi:201 [INFO] cancelling 17147981 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147981 on provider escience_a40 2024-03-17 04:08:16 proc_lsst.multi:146 [INFO] found job 17148052 in provider checkpoint proc_lsst.multi INFO: found job 17148052 in provider checkpoint 2024-03-17 04:08:16 proc_lsst.multi:201 [INFO] cancelling 17148052 on provider checkpoint proc_lsst.multi INFO: cancelling 17148052 on provider checkpoint 2024-03-17 04:08:16 proc_lsst.multi:146 [INFO] found job 17147962 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147962 in provider escience_rtx6k 2024-03-17 04:08:16 proc_lsst.multi:201 [INFO] cancelling 17147962 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17147962 on provider escience_rtx6k 2024-03-17 04:08:16 proc_lsst.multi:146 [INFO] found job 17148034 in provider checkpoint proc_lsst.multi INFO: found job 17148034 in provider checkpoint 2024-03-17 04:08:16 proc_lsst.multi:201 [INFO] cancelling 17148034 on provider checkpoint proc_lsst.multi INFO: cancelling 17148034 on provider checkpoint 2024-03-17 04:08:17 proc_lsst.multi:146 [INFO] found job 17148020 in provider checkpoint proc_lsst.multi INFO: found job 17148020 in provider checkpoint 2024-03-17 04:08:17 proc_lsst.multi:201 [INFO] cancelling 17148020 on provider checkpoint proc_lsst.multi INFO: cancelling 17148020 on provider checkpoint 2024-03-17 04:08:17 proc_lsst.multi:146 [INFO] found job 17148019 in provider checkpoint proc_lsst.multi INFO: found job 17148019 in provider checkpoint 2024-03-17 04:08:17 proc_lsst.multi:201 [INFO] cancelling 17148019 on provider checkpoint proc_lsst.multi INFO: cancelling 17148019 on provider checkpoint 2024-03-17 04:08:17 proc_lsst.multi:146 [INFO] found job 17147964 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147964 in provider escience_rtx6k 2024-03-17 04:08:17 proc_lsst.multi:201 [INFO] cancelling 17147964 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17147964 on provider escience_rtx6k 2024-03-17 04:08:18 proc_lsst.multi:146 [INFO] found job 17148022 in provider checkpoint proc_lsst.multi INFO: found job 17148022 in provider checkpoint 2024-03-17 04:08:18 proc_lsst.multi:201 [INFO] cancelling 17148022 on provider checkpoint proc_lsst.multi INFO: cancelling 17148022 on provider checkpoint 2024-03-17 04:08:18 proc_lsst.multi:146 [INFO] found job 17147960 in provider astro proc_lsst.multi INFO: found job 17147960 in provider astro 2024-03-17 04:08:18 proc_lsst.multi:201 [INFO] cancelling 17147960 on provider astro proc_lsst.multi INFO: cancelling 17147960 on provider astro 2024-03-17 04:08:18 proc_lsst.multi:146 [INFO] found job 17148001 in provider checkpoint proc_lsst.multi INFO: found job 17148001 in provider checkpoint 2024-03-17 04:08:18 proc_lsst.multi:201 [INFO] cancelling 17148001 on provider checkpoint proc_lsst.multi INFO: cancelling 17148001 on provider checkpoint 2024-03-17 04:08:18 proc_lsst.multi:146 [INFO] found job 17148047 in provider checkpoint proc_lsst.multi INFO: found job 17148047 in provider checkpoint 2024-03-17 04:08:18 proc_lsst.multi:201 [INFO] cancelling 17148047 on provider checkpoint proc_lsst.multi INFO: cancelling 17148047 on provider checkpoint 2024-03-17 04:08:19 proc_lsst.multi:146 [INFO] found job 17148048 in provider checkpoint proc_lsst.multi INFO: found job 17148048 in provider checkpoint 2024-03-17 04:08:19 proc_lsst.multi:201 [INFO] cancelling 17148048 on provider checkpoint proc_lsst.multi INFO: cancelling 17148048 on provider checkpoint 2024-03-17 04:08:19 proc_lsst.multi:146 [INFO] found job 17147966 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147966 in provider escience_rtx6k 2024-03-17 04:08:19 proc_lsst.multi:201 [INFO] cancelling 17147966 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17147966 on provider escience_rtx6k 2024-03-17 04:08:19 proc_lsst.multi:146 [INFO] found job 17147961 in provider astro proc_lsst.multi INFO: found job 17147961 in provider astro 2024-03-17 04:08:19 proc_lsst.multi:201 [INFO] cancelling 17147961 on provider astro proc_lsst.multi INFO: cancelling 17147961 on provider astro 2024-03-17 04:08:20 proc_lsst.multi:146 [INFO] found job 17148002 in provider checkpoint proc_lsst.multi INFO: found job 17148002 in provider checkpoint 2024-03-17 04:08:20 proc_lsst.multi:201 [INFO] cancelling 17148002 on provider checkpoint proc_lsst.multi INFO: cancelling 17148002 on provider checkpoint 2024-03-17 04:08:20 proc_lsst.multi:146 [INFO] found job 17147987 in provider escience_a40 proc_lsst.multi INFO: found job 17147987 in provider escience_a40 2024-03-17 04:08:20 proc_lsst.multi:201 [INFO] cancelling 17147987 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147987 on provider escience_a40 2024-03-17 04:08:20 proc_lsst.multi:146 [INFO] found job 17147979 in provider escience_a40 proc_lsst.multi INFO: found job 17147979 in provider escience_a40 2024-03-17 04:08:20 proc_lsst.multi:201 [INFO] cancelling 17147979 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147979 on provider escience_a40 2024-03-17 04:08:21 proc_lsst.multi:146 [INFO] found job 17147983 in provider escience_a40 proc_lsst.multi INFO: found job 17147983 in provider escience_a40 2024-03-17 04:08:21 proc_lsst.multi:201 [INFO] cancelling 17147983 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147983 on provider escience_a40 2024-03-17 04:08:21 proc_lsst.multi:146 [INFO] found job 17147998 in provider escience_a40 proc_lsst.multi INFO: found job 17147998 in provider escience_a40 2024-03-17 04:08:21 proc_lsst.multi:201 [INFO] cancelling 17147998 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147998 on provider escience_a40 2024-03-17 04:08:21 proc_lsst.multi:146 [INFO] found job 17147939 in provider astro proc_lsst.multi INFO: found job 17147939 in provider astro 2024-03-17 04:08:21 proc_lsst.multi:201 [INFO] cancelling 17147939 on provider astro proc_lsst.multi INFO: cancelling 17147939 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-17 04:08:21 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-17 04:08:21 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17148001', '17148002', '17148003', '17148004', '17148005', '17148006', '17148007', '17148008', '17148009', '17148010', '17148011', '17148012', '17148013', '17148014', '17148015', '17148016', '17148017', '17148018', '17148019', '17148020', '17148021', '17148022', '17148023', '17148024', '17148025', '17148026', '17148027', '17148028', '17148029', '17148030', '17148031', '17148032', '17148033', '17148034', '17148035', '17148036', '17148037', '17148038', '17148039', '17148040', '17148041', '17148042', '17148043', '17148044', '17148045', '17148046', '17148047', '17148048', '17148049', '17148050', '17148051', '17148052', '17148053', '17148054', '17148055', '17148056', '17148057', '17148058', '17148059', '17148061', '17148062', '17148063', '17148064', '17148065', '17148066', '17148067', '17148068', '17148069', '17148070', '17148071', '17148072', '17148073', '17148074', '17148075', '17148076', '17148077', '17148078', '17148079', '17148080', '17148081', '17148082', '17148083', '17148084', '17148085', '17148086', '17148087', '17148088', '17148089', '17148090', '17148091', '17148092', '17148093', '17148094', '17148095', '17148096', '17148097', '17148098', '17148099', '17148100', '17148101', '17148102', '17148103', '17148104', '17148105', '17148106', '17148107', '17148108', '17148109', '17148110', '17148111', '17148112', '17148113', '17148114', '17148115', '17148116', '17148117', '17148118', '17148119', '17148120', '17148121', '17148122', '17148123', '17148124', '17148125', '17148126', '17148127', '17148128', '17148129', '17148130', '17148131', '17148132', '17148133', '17148134', '17148135', '17148136', '17148137', '17148138', '17148139', '17148140', '17148141', '17148142', '17148143', '17148144', '17148145', '17148146', '17148147', '17148148', '17148149', '17148150', '17148151', '17148152', '17148153', '17148154', '17148155', '17148156', '17148157', '17148158', '17148159', '17148160', '17148161', '17148162', '17148163', '17148166', '17148167', '17148168', '17148169', '17148170', '17148171', '17148172', '17148173', '17148174', '17148175', '17148176', '17148177', '17148178', '17148179', '17148180', '17148181', '17148182', '17148183', '17148184', '17148185', '17148186', '17148187', '17148188', '17148189', '17148190', '17148191', '17148192', '17148193', '17148194', '17148195', '17148196', '17148197', '17148198', '17148199', '17148200', '17148201', '17148202', '17148203', '17148204', '17148205', '17148206', '17148207', '17148208', '17148209', '17148210', '17148211', '17148212', '17148213', '17148214', '17148215', '17148216', '17148217', '17148218', '17148219', '17148220', '17148221', '17148222', '17148223', '17148224', '17148225', '17148226', '17148227', '17148228', '17148229', '17148230', '17148231', '17148232', '17148233', '17148234', '17148235', '17148236', '17148237', '17148238', '17148239', '17148240', '17148241', '17148242', '17148243', '17148244', '17148245', '17148246', '17148247', '17148248', '17148249', '17148250', '17148251', '17148252', '17148253', '17148254', '17148255', '17148256', '17148257', '17148258', '17147931', '17147932', '17147933', '17147934', '17147935', '17147936', '17147937', '17147938', '17147939', '17147940', '17147941', '17147942', '17147943', '17147944', '17147945', '17147946', '17147947', '17147948', '17147950', '17147951', '17147952', '17147953', '17147954', '17147955', '17147956', '17147957', '17147958', '17147959', '17147960', '17147961', '17147977', '17147978', '17147979', '17147980', '17147981', '17147983', '17147984', '17147985', '17147986', '17147987', '17147988', '17147989', '17147992', '17147993', '17147994', '17147995', '17147996', '17147997', '17147998', '17148000', '17147962', '17147963', '17147964', '17147965', '17147966', '3544'] proc_lsst.multi INFO: new jobs since last cancel ['17148001', '17148002', '17148003', '17148004', '17148005', '17148006', '17148007', '17148008', '17148009', '17148010', '17148011', '17148012', '17148013', '17148014', '17148015', '17148016', '17148017', '17148018', '17148019', '17148020', '17148021', '17148022', '17148023', '17148024', '17148025', '17148026', '17148027', '17148028', '17148029', '17148030', '17148031', '17148032', '17148033', '17148034', '17148035', '17148036', '17148037', '17148038', '17148039', '17148040', '17148041', '17148042', '17148043', '17148044', '17148045', '17148046', '17148047', '17148048', '17148049', '17148050', '17148051', '17148052', '17148053', '17148054', '17148055', '17148056', '17148057', '17148058', '17148059', '17148061', '17148062', '17148063', '17148064', '17148065', '17148066', '17148067', '17148068', '17148069', '17148070', '17148071', '17148072', '17148073', '17148074', '17148075', '17148076', '17148077', '17148078', '17148079', '17148080', '17148081', '17148082', '17148083', '17148084', '17148085', '17148086', '17148087', '17148088', '17148089', '17148090', '17148091', '17148092', '17148093', '17148094', '17148095', '17148096', '17148097', '17148098', '17148099', '17148100', '17148101', '17148102', '17148103', '17148104', '17148105', '17148106', '17148107', '17148108', '17148109', '17148110', '17148111', '17148112', '17148113', '17148114', '17148115', '17148116', '17148117', '17148118', '17148119', '17148120', '17148121', '17148122', '17148123', '17148124', '17148125', '17148126', '17148127', '17148128', '17148129', '17148130', '17148131', '17148132', '17148133', '17148134', '17148135', '17148136', '17148137', '17148138', '17148139', '17148140', '17148141', '17148142', '17148143', '17148144', '17148145', '17148146', '17148147', '17148148', '17148149', '17148150', '17148151', '17148152', '17148153', '17148154', '17148155', '17148156', '17148157', '17148158', '17148159', '17148160', '17148161', '17148162', '17148163', '17148166', '17148167', '17148168', '17148169', '17148170', '17148171', '17148172', '17148173', '17148174', '17148175', '17148176', '17148177', '17148178', '17148179', '17148180', '17148181', '17148182', '17148183', '17148184', '17148185', '17148186', '17148187', '17148188', '17148189', '17148190', '17148191', '17148192', '17148193', '17148194', '17148195', '17148196', '17148197', '17148198', '17148199', '17148200', '17148201', '17148202', '17148203', '17148204', '17148205', '17148206', '17148207', '17148208', '17148209', '17148210', '17148211', '17148212', '17148213', '17148214', '17148215', '17148216', '17148217', '17148218', '17148219', '17148220', '17148221', '17148222', '17148223', '17148224', '17148225', '17148226', '17148227', '17148228', '17148229', '17148230', '17148231', '17148232', '17148233', '17148234', '17148235', '17148236', '17148237', '17148238', '17148239', '17148240', '17148241', '17148242', '17148243', '17148244', '17148245', '17148246', '17148247', '17148248', '17148249', '17148250', '17148251', '17148252', '17148253', '17148254', '17148255', '17148256', '17148257', '17148258', '17147931', '17147932', '17147933', '17147934', '17147935', '17147936', '17147937', '17147938', '17147939', '17147940', '17147941', '17147942', '17147943', '17147944', '17147945', '17147946', '17147947', '17147948', '17147950', '17147951', '17147952', '17147953', '17147954', '17147955', '17147956', '17147957', '17147958', '17147959', '17147960', '17147961', '17147977', '17147978', '17147979', '17147980', '17147981', '17147983', '17147984', '17147985', '17147986', '17147987', '17147988', '17147989', '17147992', '17147993', '17147994', '17147995', '17147996', '17147997', '17147998', '17148000', '17147962', '17147963', '17147964', '17147965', '17147966', '3544'] 2024-03-17 04:08:21 proc_lsst.multi:146 [INFO] found job 17148001 in provider checkpoint proc_lsst.multi INFO: found job 17148001 in provider checkpoint 2024-03-17 04:08:21 proc_lsst.multi:201 [INFO] cancelling 17148001 on provider checkpoint proc_lsst.multi INFO: cancelling 17148001 on provider checkpoint 2024-03-17 04:08:22 proc_lsst.multi:146 [INFO] found job 17148002 in provider checkpoint proc_lsst.multi INFO: found job 17148002 in provider checkpoint 2024-03-17 04:08:22 proc_lsst.multi:201 [INFO] cancelling 17148002 on provider checkpoint proc_lsst.multi INFO: cancelling 17148002 on provider checkpoint 2024-03-17 04:08:22 proc_lsst.multi:146 [INFO] found job 17148003 in provider checkpoint proc_lsst.multi INFO: found job 17148003 in provider checkpoint 2024-03-17 04:08:22 proc_lsst.multi:201 [INFO] cancelling 17148003 on provider checkpoint proc_lsst.multi INFO: cancelling 17148003 on provider checkpoint 2024-03-17 04:08:22 proc_lsst.multi:146 [INFO] found job 17148004 in provider checkpoint proc_lsst.multi INFO: found job 17148004 in provider checkpoint 2024-03-17 04:08:22 proc_lsst.multi:201 [INFO] cancelling 17148004 on provider checkpoint proc_lsst.multi INFO: cancelling 17148004 on provider checkpoint 2024-03-17 04:08:23 proc_lsst.multi:146 [INFO] found job 17148005 in provider checkpoint proc_lsst.multi INFO: found job 17148005 in provider checkpoint 2024-03-17 04:08:23 proc_lsst.multi:201 [INFO] cancelling 17148005 on provider checkpoint proc_lsst.multi INFO: cancelling 17148005 on provider checkpoint 2024-03-17 04:08:23 proc_lsst.multi:146 [INFO] found job 17148006 in provider checkpoint proc_lsst.multi INFO: found job 17148006 in provider checkpoint 2024-03-17 04:08:23 proc_lsst.multi:201 [INFO] cancelling 17148006 on provider checkpoint proc_lsst.multi INFO: cancelling 17148006 on provider checkpoint 2024-03-17 04:08:23 proc_lsst.multi:146 [INFO] found job 17148007 in provider checkpoint proc_lsst.multi INFO: found job 17148007 in provider checkpoint 2024-03-17 04:08:23 proc_lsst.multi:201 [INFO] cancelling 17148007 on provider checkpoint proc_lsst.multi INFO: cancelling 17148007 on provider checkpoint 2024-03-17 04:08:23 proc_lsst.multi:146 [INFO] found job 17148008 in provider checkpoint proc_lsst.multi INFO: found job 17148008 in provider checkpoint 2024-03-17 04:08:23 proc_lsst.multi:201 [INFO] cancelling 17148008 on provider checkpoint proc_lsst.multi INFO: cancelling 17148008 on provider checkpoint 2024-03-17 04:08:24 proc_lsst.multi:146 [INFO] found job 17148009 in provider checkpoint proc_lsst.multi INFO: found job 17148009 in provider checkpoint 2024-03-17 04:08:24 proc_lsst.multi:201 [INFO] cancelling 17148009 on provider checkpoint proc_lsst.multi INFO: cancelling 17148009 on provider checkpoint 2024-03-17 04:08:24 proc_lsst.multi:146 [INFO] found job 17148010 in provider checkpoint proc_lsst.multi INFO: found job 17148010 in provider checkpoint 2024-03-17 04:08:24 proc_lsst.multi:201 [INFO] cancelling 17148010 on provider checkpoint proc_lsst.multi INFO: cancelling 17148010 on provider checkpoint 2024-03-17 04:08:24 proc_lsst.multi:146 [INFO] found job 17148011 in provider checkpoint proc_lsst.multi INFO: found job 17148011 in provider checkpoint 2024-03-17 04:08:24 proc_lsst.multi:201 [INFO] cancelling 17148011 on provider checkpoint proc_lsst.multi INFO: cancelling 17148011 on provider checkpoint 2024-03-17 04:08:25 proc_lsst.multi:146 [INFO] found job 17148012 in provider checkpoint proc_lsst.multi INFO: found job 17148012 in provider checkpoint 2024-03-17 04:08:25 proc_lsst.multi:201 [INFO] cancelling 17148012 on provider checkpoint proc_lsst.multi INFO: cancelling 17148012 on provider checkpoint 2024-03-17 04:08:25 proc_lsst.multi:146 [INFO] found job 17148013 in provider checkpoint proc_lsst.multi INFO: found job 17148013 in provider checkpoint 2024-03-17 04:08:25 proc_lsst.multi:201 [INFO] cancelling 17148013 on provider checkpoint proc_lsst.multi INFO: cancelling 17148013 on provider checkpoint 2024-03-17 04:08:25 proc_lsst.multi:146 [INFO] found job 17148014 in provider checkpoint proc_lsst.multi INFO: found job 17148014 in provider checkpoint 2024-03-17 04:08:25 proc_lsst.multi:201 [INFO] cancelling 17148014 on provider checkpoint proc_lsst.multi INFO: cancelling 17148014 on provider checkpoint 2024-03-17 04:08:26 proc_lsst.multi:146 [INFO] found job 17148015 in provider checkpoint proc_lsst.multi INFO: found job 17148015 in provider checkpoint 2024-03-17 04:08:26 proc_lsst.multi:201 [INFO] cancelling 17148015 on provider checkpoint proc_lsst.multi INFO: cancelling 17148015 on provider checkpoint 2024-03-17 04:08:26 proc_lsst.multi:146 [INFO] found job 17148016 in provider checkpoint proc_lsst.multi INFO: found job 17148016 in provider checkpoint 2024-03-17 04:08:26 proc_lsst.multi:201 [INFO] cancelling 17148016 on provider checkpoint proc_lsst.multi INFO: cancelling 17148016 on provider checkpoint 2024-03-17 04:08:26 proc_lsst.multi:146 [INFO] found job 17148017 in provider checkpoint proc_lsst.multi INFO: found job 17148017 in provider checkpoint 2024-03-17 04:08:26 proc_lsst.multi:201 [INFO] cancelling 17148017 on provider checkpoint proc_lsst.multi INFO: cancelling 17148017 on provider checkpoint 2024-03-17 04:08:26 proc_lsst.multi:146 [INFO] found job 17148018 in provider checkpoint proc_lsst.multi INFO: found job 17148018 in provider checkpoint 2024-03-17 04:08:26 proc_lsst.multi:201 [INFO] cancelling 17148018 on provider checkpoint proc_lsst.multi INFO: cancelling 17148018 on provider checkpoint 2024-03-17 04:08:27 proc_lsst.multi:146 [INFO] found job 17148019 in provider checkpoint proc_lsst.multi INFO: found job 17148019 in provider checkpoint 2024-03-17 04:08:27 proc_lsst.multi:201 [INFO] cancelling 17148019 on provider checkpoint proc_lsst.multi INFO: cancelling 17148019 on provider checkpoint 2024-03-17 04:08:27 proc_lsst.multi:146 [INFO] found job 17148020 in provider checkpoint proc_lsst.multi INFO: found job 17148020 in provider checkpoint 2024-03-17 04:08:27 proc_lsst.multi:201 [INFO] cancelling 17148020 on provider checkpoint proc_lsst.multi INFO: cancelling 17148020 on provider checkpoint 2024-03-17 04:08:27 proc_lsst.multi:146 [INFO] found job 17148021 in provider checkpoint proc_lsst.multi INFO: found job 17148021 in provider checkpoint 2024-03-17 04:08:27 proc_lsst.multi:201 [INFO] cancelling 17148021 on provider checkpoint proc_lsst.multi INFO: cancelling 17148021 on provider checkpoint 2024-03-17 04:08:28 proc_lsst.multi:146 [INFO] found job 17148022 in provider checkpoint proc_lsst.multi INFO: found job 17148022 in provider checkpoint 2024-03-17 04:08:28 proc_lsst.multi:201 [INFO] cancelling 17148022 on provider checkpoint proc_lsst.multi INFO: cancelling 17148022 on provider checkpoint 2024-03-17 04:08:28 proc_lsst.multi:146 [INFO] found job 17148023 in provider checkpoint proc_lsst.multi INFO: found job 17148023 in provider checkpoint 2024-03-17 04:08:28 proc_lsst.multi:201 [INFO] cancelling 17148023 on provider checkpoint proc_lsst.multi INFO: cancelling 17148023 on provider checkpoint 2024-03-17 04:08:28 proc_lsst.multi:146 [INFO] found job 17148024 in provider checkpoint proc_lsst.multi INFO: found job 17148024 in provider checkpoint 2024-03-17 04:08:28 proc_lsst.multi:201 [INFO] cancelling 17148024 on provider checkpoint proc_lsst.multi INFO: cancelling 17148024 on provider checkpoint 2024-03-17 04:08:28 proc_lsst.multi:146 [INFO] found job 17148025 in provider checkpoint proc_lsst.multi INFO: found job 17148025 in provider checkpoint 2024-03-17 04:08:28 proc_lsst.multi:201 [INFO] cancelling 17148025 on provider checkpoint proc_lsst.multi INFO: cancelling 17148025 on provider checkpoint 2024-03-17 04:08:29 proc_lsst.multi:146 [INFO] found job 17148026 in provider checkpoint proc_lsst.multi INFO: found job 17148026 in provider checkpoint 2024-03-17 04:08:29 proc_lsst.multi:201 [INFO] cancelling 17148026 on provider checkpoint proc_lsst.multi INFO: cancelling 17148026 on provider checkpoint 2024-03-17 04:08:29 proc_lsst.multi:146 [INFO] found job 17148027 in provider checkpoint proc_lsst.multi INFO: found job 17148027 in provider checkpoint 2024-03-17 04:08:29 proc_lsst.multi:201 [INFO] cancelling 17148027 on provider checkpoint proc_lsst.multi INFO: cancelling 17148027 on provider checkpoint 2024-03-17 04:08:29 proc_lsst.multi:146 [INFO] found job 17148028 in provider checkpoint proc_lsst.multi INFO: found job 17148028 in provider checkpoint 2024-03-17 04:08:29 proc_lsst.multi:201 [INFO] cancelling 17148028 on provider checkpoint proc_lsst.multi INFO: cancelling 17148028 on provider checkpoint 2024-03-17 04:08:30 proc_lsst.multi:146 [INFO] found job 17148029 in provider checkpoint proc_lsst.multi INFO: found job 17148029 in provider checkpoint 2024-03-17 04:08:30 proc_lsst.multi:201 [INFO] cancelling 17148029 on provider checkpoint proc_lsst.multi INFO: cancelling 17148029 on provider checkpoint 2024-03-17 04:08:30 proc_lsst.multi:146 [INFO] found job 17148030 in provider checkpoint proc_lsst.multi INFO: found job 17148030 in provider checkpoint 2024-03-17 04:08:30 proc_lsst.multi:201 [INFO] cancelling 17148030 on provider checkpoint proc_lsst.multi INFO: cancelling 17148030 on provider checkpoint 2024-03-17 04:08:30 proc_lsst.multi:146 [INFO] found job 17148031 in provider checkpoint proc_lsst.multi INFO: found job 17148031 in provider checkpoint 2024-03-17 04:08:30 proc_lsst.multi:201 [INFO] cancelling 17148031 on provider checkpoint proc_lsst.multi INFO: cancelling 17148031 on provider checkpoint 2024-03-17 04:08:31 proc_lsst.multi:146 [INFO] found job 17148032 in provider checkpoint proc_lsst.multi INFO: found job 17148032 in provider checkpoint 2024-03-17 04:08:31 proc_lsst.multi:201 [INFO] cancelling 17148032 on provider checkpoint proc_lsst.multi INFO: cancelling 17148032 on provider checkpoint 2024-03-17 04:08:31 proc_lsst.multi:146 [INFO] found job 17148033 in provider checkpoint proc_lsst.multi INFO: found job 17148033 in provider checkpoint 2024-03-17 04:08:31 proc_lsst.multi:201 [INFO] cancelling 17148033 on provider checkpoint proc_lsst.multi INFO: cancelling 17148033 on provider checkpoint 2024-03-17 04:08:31 proc_lsst.multi:146 [INFO] found job 17148034 in provider checkpoint proc_lsst.multi INFO: found job 17148034 in provider checkpoint 2024-03-17 04:08:31 proc_lsst.multi:201 [INFO] cancelling 17148034 on provider checkpoint proc_lsst.multi INFO: cancelling 17148034 on provider checkpoint 2024-03-17 04:08:32 proc_lsst.multi:146 [INFO] found job 17148035 in provider checkpoint proc_lsst.multi INFO: found job 17148035 in provider checkpoint 2024-03-17 04:08:32 proc_lsst.multi:201 [INFO] cancelling 17148035 on provider checkpoint proc_lsst.multi INFO: cancelling 17148035 on provider checkpoint 2024-03-17 04:08:32 proc_lsst.multi:146 [INFO] found job 17148036 in provider checkpoint proc_lsst.multi INFO: found job 17148036 in provider checkpoint 2024-03-17 04:08:32 proc_lsst.multi:201 [INFO] cancelling 17148036 on provider checkpoint proc_lsst.multi INFO: cancelling 17148036 on provider checkpoint 2024-03-17 04:08:32 proc_lsst.multi:146 [INFO] found job 17148037 in provider checkpoint proc_lsst.multi INFO: found job 17148037 in provider checkpoint 2024-03-17 04:08:32 proc_lsst.multi:201 [INFO] cancelling 17148037 on provider checkpoint proc_lsst.multi INFO: cancelling 17148037 on provider checkpoint 2024-03-17 04:08:33 proc_lsst.multi:146 [INFO] found job 17148038 in provider checkpoint proc_lsst.multi INFO: found job 17148038 in provider checkpoint 2024-03-17 04:08:33 proc_lsst.multi:201 [INFO] cancelling 17148038 on provider checkpoint proc_lsst.multi INFO: cancelling 17148038 on provider checkpoint 2024-03-17 04:08:33 proc_lsst.multi:146 [INFO] found job 17148039 in provider checkpoint proc_lsst.multi INFO: found job 17148039 in provider checkpoint 2024-03-17 04:08:33 proc_lsst.multi:201 [INFO] cancelling 17148039 on provider checkpoint proc_lsst.multi INFO: cancelling 17148039 on provider checkpoint 2024-03-17 04:08:33 proc_lsst.multi:146 [INFO] found job 17148040 in provider checkpoint proc_lsst.multi INFO: found job 17148040 in provider checkpoint 2024-03-17 04:08:33 proc_lsst.multi:201 [INFO] cancelling 17148040 on provider checkpoint proc_lsst.multi INFO: cancelling 17148040 on provider checkpoint 2024-03-17 04:08:34 proc_lsst.multi:146 [INFO] found job 17148041 in provider checkpoint proc_lsst.multi INFO: found job 17148041 in provider checkpoint 2024-03-17 04:08:34 proc_lsst.multi:201 [INFO] cancelling 17148041 on provider checkpoint proc_lsst.multi INFO: cancelling 17148041 on provider checkpoint 2024-03-17 04:08:34 proc_lsst.multi:146 [INFO] found job 17148042 in provider checkpoint proc_lsst.multi INFO: found job 17148042 in provider checkpoint 2024-03-17 04:08:34 proc_lsst.multi:201 [INFO] cancelling 17148042 on provider checkpoint proc_lsst.multi INFO: cancelling 17148042 on provider checkpoint 2024-03-17 04:08:34 proc_lsst.multi:146 [INFO] found job 17148043 in provider checkpoint proc_lsst.multi INFO: found job 17148043 in provider checkpoint 2024-03-17 04:08:34 proc_lsst.multi:201 [INFO] cancelling 17148043 on provider checkpoint proc_lsst.multi INFO: cancelling 17148043 on provider checkpoint 2024-03-17 04:08:34 proc_lsst.multi:146 [INFO] found job 17148044 in provider checkpoint proc_lsst.multi INFO: found job 17148044 in provider checkpoint 2024-03-17 04:08:34 proc_lsst.multi:201 [INFO] cancelling 17148044 on provider checkpoint proc_lsst.multi INFO: cancelling 17148044 on provider checkpoint 2024-03-17 04:08:34 proc_lsst.multi:146 [INFO] found job 17148045 in provider checkpoint proc_lsst.multi INFO: found job 17148045 in provider checkpoint 2024-03-17 04:08:34 proc_lsst.multi:201 [INFO] cancelling 17148045 on provider checkpoint proc_lsst.multi INFO: cancelling 17148045 on provider checkpoint 2024-03-17 04:08:35 proc_lsst.multi:146 [INFO] found job 17148046 in provider checkpoint proc_lsst.multi INFO: found job 17148046 in provider checkpoint 2024-03-17 04:08:35 proc_lsst.multi:201 [INFO] cancelling 17148046 on provider checkpoint proc_lsst.multi INFO: cancelling 17148046 on provider checkpoint 2024-03-17 04:08:35 proc_lsst.multi:146 [INFO] found job 17148047 in provider checkpoint proc_lsst.multi INFO: found job 17148047 in provider checkpoint 2024-03-17 04:08:35 proc_lsst.multi:201 [INFO] cancelling 17148047 on provider checkpoint proc_lsst.multi INFO: cancelling 17148047 on provider checkpoint 2024-03-17 04:08:35 proc_lsst.multi:146 [INFO] found job 17148048 in provider checkpoint proc_lsst.multi INFO: found job 17148048 in provider checkpoint 2024-03-17 04:08:35 proc_lsst.multi:201 [INFO] cancelling 17148048 on provider checkpoint proc_lsst.multi INFO: cancelling 17148048 on provider checkpoint 2024-03-17 04:08:35 proc_lsst.multi:146 [INFO] found job 17148049 in provider checkpoint proc_lsst.multi INFO: found job 17148049 in provider checkpoint 2024-03-17 04:08:35 proc_lsst.multi:201 [INFO] cancelling 17148049 on provider checkpoint proc_lsst.multi INFO: cancelling 17148049 on provider checkpoint 2024-03-17 04:08:36 proc_lsst.multi:146 [INFO] found job 17148050 in provider checkpoint proc_lsst.multi INFO: found job 17148050 in provider checkpoint 2024-03-17 04:08:36 proc_lsst.multi:201 [INFO] cancelling 17148050 on provider checkpoint proc_lsst.multi INFO: cancelling 17148050 on provider checkpoint 2024-03-17 04:08:36 proc_lsst.multi:146 [INFO] found job 17148051 in provider checkpoint proc_lsst.multi INFO: found job 17148051 in provider checkpoint 2024-03-17 04:08:36 proc_lsst.multi:201 [INFO] cancelling 17148051 on provider checkpoint proc_lsst.multi INFO: cancelling 17148051 on provider checkpoint 2024-03-17 04:08:36 proc_lsst.multi:146 [INFO] found job 17148052 in provider checkpoint proc_lsst.multi INFO: found job 17148052 in provider checkpoint 2024-03-17 04:08:36 proc_lsst.multi:201 [INFO] cancelling 17148052 on provider checkpoint proc_lsst.multi INFO: cancelling 17148052 on provider checkpoint 2024-03-17 04:08:37 proc_lsst.multi:146 [INFO] found job 17148053 in provider checkpoint proc_lsst.multi INFO: found job 17148053 in provider checkpoint 2024-03-17 04:08:37 proc_lsst.multi:201 [INFO] cancelling 17148053 on provider checkpoint proc_lsst.multi INFO: cancelling 17148053 on provider checkpoint 2024-03-17 04:08:37 proc_lsst.multi:146 [INFO] found job 17148054 in provider checkpoint proc_lsst.multi INFO: found job 17148054 in provider checkpoint 2024-03-17 04:08:37 proc_lsst.multi:201 [INFO] cancelling 17148054 on provider checkpoint proc_lsst.multi INFO: cancelling 17148054 on provider checkpoint 2024-03-17 04:08:37 proc_lsst.multi:146 [INFO] found job 17148055 in provider checkpoint proc_lsst.multi INFO: found job 17148055 in provider checkpoint 2024-03-17 04:08:37 proc_lsst.multi:201 [INFO] cancelling 17148055 on provider checkpoint proc_lsst.multi INFO: cancelling 17148055 on provider checkpoint 2024-03-17 04:08:38 proc_lsst.multi:146 [INFO] found job 17148056 in provider checkpoint proc_lsst.multi INFO: found job 17148056 in provider checkpoint 2024-03-17 04:08:38 proc_lsst.multi:201 [INFO] cancelling 17148056 on provider checkpoint proc_lsst.multi INFO: cancelling 17148056 on provider checkpoint 2024-03-17 04:08:38 proc_lsst.multi:146 [INFO] found job 17148057 in provider checkpoint proc_lsst.multi INFO: found job 17148057 in provider checkpoint 2024-03-17 04:08:38 proc_lsst.multi:201 [INFO] cancelling 17148057 on provider checkpoint proc_lsst.multi INFO: cancelling 17148057 on provider checkpoint 2024-03-17 04:08:38 proc_lsst.multi:146 [INFO] found job 17148058 in provider checkpoint proc_lsst.multi INFO: found job 17148058 in provider checkpoint 2024-03-17 04:08:38 proc_lsst.multi:201 [INFO] cancelling 17148058 on provider checkpoint proc_lsst.multi INFO: cancelling 17148058 on provider checkpoint 2024-03-17 04:08:38 proc_lsst.multi:146 [INFO] found job 17148059 in provider checkpoint proc_lsst.multi INFO: found job 17148059 in provider checkpoint 2024-03-17 04:08:38 proc_lsst.multi:201 [INFO] cancelling 17148059 on provider checkpoint proc_lsst.multi INFO: cancelling 17148059 on provider checkpoint 2024-03-17 04:08:39 proc_lsst.multi:146 [INFO] found job 17148061 in provider checkpoint proc_lsst.multi INFO: found job 17148061 in provider checkpoint 2024-03-17 04:08:39 proc_lsst.multi:201 [INFO] cancelling 17148061 on provider checkpoint proc_lsst.multi INFO: cancelling 17148061 on provider checkpoint 2024-03-17 04:08:39 proc_lsst.multi:146 [INFO] found job 17148062 in provider checkpoint proc_lsst.multi INFO: found job 17148062 in provider checkpoint 2024-03-17 04:08:39 proc_lsst.multi:201 [INFO] cancelling 17148062 on provider checkpoint proc_lsst.multi INFO: cancelling 17148062 on provider checkpoint 2024-03-17 04:08:39 proc_lsst.multi:146 [INFO] found job 17148063 in provider checkpoint proc_lsst.multi INFO: found job 17148063 in provider checkpoint 2024-03-17 04:08:39 proc_lsst.multi:201 [INFO] cancelling 17148063 on provider checkpoint proc_lsst.multi INFO: cancelling 17148063 on provider checkpoint 2024-03-17 04:08:39 proc_lsst.multi:146 [INFO] found job 17148064 in provider checkpoint proc_lsst.multi INFO: found job 17148064 in provider checkpoint 2024-03-17 04:08:39 proc_lsst.multi:201 [INFO] cancelling 17148064 on provider checkpoint proc_lsst.multi INFO: cancelling 17148064 on provider checkpoint 2024-03-17 04:08:40 proc_lsst.multi:146 [INFO] found job 17148065 in provider checkpoint proc_lsst.multi INFO: found job 17148065 in provider checkpoint 2024-03-17 04:08:40 proc_lsst.multi:201 [INFO] cancelling 17148065 on provider checkpoint proc_lsst.multi INFO: cancelling 17148065 on provider checkpoint 2024-03-17 04:08:40 proc_lsst.multi:146 [INFO] found job 17148066 in provider checkpoint proc_lsst.multi INFO: found job 17148066 in provider checkpoint 2024-03-17 04:08:40 proc_lsst.multi:201 [INFO] cancelling 17148066 on provider checkpoint proc_lsst.multi INFO: cancelling 17148066 on provider checkpoint 2024-03-17 04:08:40 proc_lsst.multi:146 [INFO] found job 17148067 in provider checkpoint proc_lsst.multi INFO: found job 17148067 in provider checkpoint 2024-03-17 04:08:40 proc_lsst.multi:201 [INFO] cancelling 17148067 on provider checkpoint proc_lsst.multi INFO: cancelling 17148067 on provider checkpoint 2024-03-17 04:08:41 proc_lsst.multi:146 [INFO] found job 17148068 in provider checkpoint proc_lsst.multi INFO: found job 17148068 in provider checkpoint 2024-03-17 04:08:41 proc_lsst.multi:201 [INFO] cancelling 17148068 on provider checkpoint proc_lsst.multi INFO: cancelling 17148068 on provider checkpoint 2024-03-17 04:08:41 proc_lsst.multi:146 [INFO] found job 17148069 in provider checkpoint proc_lsst.multi INFO: found job 17148069 in provider checkpoint 2024-03-17 04:08:41 proc_lsst.multi:201 [INFO] cancelling 17148069 on provider checkpoint proc_lsst.multi INFO: cancelling 17148069 on provider checkpoint 2024-03-17 04:08:41 proc_lsst.multi:146 [INFO] found job 17148070 in provider checkpoint proc_lsst.multi INFO: found job 17148070 in provider checkpoint 2024-03-17 04:08:41 proc_lsst.multi:201 [INFO] cancelling 17148070 on provider checkpoint proc_lsst.multi INFO: cancelling 17148070 on provider checkpoint 2024-03-17 04:08:42 proc_lsst.multi:146 [INFO] found job 17148071 in provider checkpoint proc_lsst.multi INFO: found job 17148071 in provider checkpoint 2024-03-17 04:08:42 proc_lsst.multi:201 [INFO] cancelling 17148071 on provider checkpoint proc_lsst.multi INFO: cancelling 17148071 on provider checkpoint 2024-03-17 04:08:42 proc_lsst.multi:146 [INFO] found job 17148072 in provider checkpoint proc_lsst.multi INFO: found job 17148072 in provider checkpoint 2024-03-17 04:08:42 proc_lsst.multi:201 [INFO] cancelling 17148072 on provider checkpoint proc_lsst.multi INFO: cancelling 17148072 on provider checkpoint 2024-03-17 04:08:42 proc_lsst.multi:146 [INFO] found job 17148073 in provider checkpoint proc_lsst.multi INFO: found job 17148073 in provider checkpoint 2024-03-17 04:08:42 proc_lsst.multi:201 [INFO] cancelling 17148073 on provider checkpoint proc_lsst.multi INFO: cancelling 17148073 on provider checkpoint 2024-03-17 04:08:43 proc_lsst.multi:146 [INFO] found job 17148074 in provider checkpoint proc_lsst.multi INFO: found job 17148074 in provider checkpoint 2024-03-17 04:08:43 proc_lsst.multi:201 [INFO] cancelling 17148074 on provider checkpoint proc_lsst.multi INFO: cancelling 17148074 on provider checkpoint 2024-03-17 04:08:43 proc_lsst.multi:146 [INFO] found job 17148075 in provider checkpoint proc_lsst.multi INFO: found job 17148075 in provider checkpoint 2024-03-17 04:08:43 proc_lsst.multi:201 [INFO] cancelling 17148075 on provider checkpoint proc_lsst.multi INFO: cancelling 17148075 on provider checkpoint 2024-03-17 04:08:43 proc_lsst.multi:146 [INFO] found job 17148076 in provider checkpoint proc_lsst.multi INFO: found job 17148076 in provider checkpoint 2024-03-17 04:08:43 proc_lsst.multi:201 [INFO] cancelling 17148076 on provider checkpoint proc_lsst.multi INFO: cancelling 17148076 on provider checkpoint 2024-03-17 04:08:44 proc_lsst.multi:146 [INFO] found job 17148077 in provider checkpoint proc_lsst.multi INFO: found job 17148077 in provider checkpoint 2024-03-17 04:08:44 proc_lsst.multi:201 [INFO] cancelling 17148077 on provider checkpoint proc_lsst.multi INFO: cancelling 17148077 on provider checkpoint 2024-03-17 04:08:44 proc_lsst.multi:146 [INFO] found job 17148078 in provider checkpoint proc_lsst.multi INFO: found job 17148078 in provider checkpoint 2024-03-17 04:08:44 proc_lsst.multi:201 [INFO] cancelling 17148078 on provider checkpoint proc_lsst.multi INFO: cancelling 17148078 on provider checkpoint 2024-03-17 04:08:44 proc_lsst.multi:146 [INFO] found job 17148079 in provider checkpoint proc_lsst.multi INFO: found job 17148079 in provider checkpoint 2024-03-17 04:08:44 proc_lsst.multi:201 [INFO] cancelling 17148079 on provider checkpoint proc_lsst.multi INFO: cancelling 17148079 on provider checkpoint 2024-03-17 04:08:44 proc_lsst.multi:146 [INFO] found job 17148080 in provider checkpoint proc_lsst.multi INFO: found job 17148080 in provider checkpoint 2024-03-17 04:08:44 proc_lsst.multi:201 [INFO] cancelling 17148080 on provider checkpoint proc_lsst.multi INFO: cancelling 17148080 on provider checkpoint 2024-03-17 04:08:45 proc_lsst.multi:146 [INFO] found job 17148081 in provider checkpoint proc_lsst.multi INFO: found job 17148081 in provider checkpoint 2024-03-17 04:08:45 proc_lsst.multi:201 [INFO] cancelling 17148081 on provider checkpoint proc_lsst.multi INFO: cancelling 17148081 on provider checkpoint 2024-03-17 04:08:45 proc_lsst.multi:146 [INFO] found job 17148082 in provider checkpoint proc_lsst.multi INFO: found job 17148082 in provider checkpoint 2024-03-17 04:08:45 proc_lsst.multi:201 [INFO] cancelling 17148082 on provider checkpoint proc_lsst.multi INFO: cancelling 17148082 on provider checkpoint 2024-03-17 04:08:45 proc_lsst.multi:146 [INFO] found job 17148083 in provider checkpoint proc_lsst.multi INFO: found job 17148083 in provider checkpoint 2024-03-17 04:08:45 proc_lsst.multi:201 [INFO] cancelling 17148083 on provider checkpoint proc_lsst.multi INFO: cancelling 17148083 on provider checkpoint 2024-03-17 04:08:46 proc_lsst.multi:146 [INFO] found job 17148084 in provider checkpoint proc_lsst.multi INFO: found job 17148084 in provider checkpoint 2024-03-17 04:08:46 proc_lsst.multi:201 [INFO] cancelling 17148084 on provider checkpoint proc_lsst.multi INFO: cancelling 17148084 on provider checkpoint 2024-03-17 04:08:46 proc_lsst.multi:146 [INFO] found job 17148085 in provider checkpoint proc_lsst.multi INFO: found job 17148085 in provider checkpoint 2024-03-17 04:08:46 proc_lsst.multi:201 [INFO] cancelling 17148085 on provider checkpoint proc_lsst.multi INFO: cancelling 17148085 on provider checkpoint 2024-03-17 04:08:46 proc_lsst.multi:146 [INFO] found job 17148086 in provider checkpoint proc_lsst.multi INFO: found job 17148086 in provider checkpoint 2024-03-17 04:08:46 proc_lsst.multi:201 [INFO] cancelling 17148086 on provider checkpoint proc_lsst.multi INFO: cancelling 17148086 on provider checkpoint 2024-03-17 04:08:46 proc_lsst.multi:146 [INFO] found job 17148087 in provider checkpoint proc_lsst.multi INFO: found job 17148087 in provider checkpoint 2024-03-17 04:08:46 proc_lsst.multi:201 [INFO] cancelling 17148087 on provider checkpoint proc_lsst.multi INFO: cancelling 17148087 on provider checkpoint 2024-03-17 04:08:47 proc_lsst.multi:146 [INFO] found job 17148088 in provider checkpoint proc_lsst.multi INFO: found job 17148088 in provider checkpoint 2024-03-17 04:08:47 proc_lsst.multi:201 [INFO] cancelling 17148088 on provider checkpoint proc_lsst.multi INFO: cancelling 17148088 on provider checkpoint 2024-03-17 04:08:47 proc_lsst.multi:146 [INFO] found job 17148089 in provider checkpoint proc_lsst.multi INFO: found job 17148089 in provider checkpoint 2024-03-17 04:08:47 proc_lsst.multi:201 [INFO] cancelling 17148089 on provider checkpoint proc_lsst.multi INFO: cancelling 17148089 on provider checkpoint 2024-03-17 04:08:47 proc_lsst.multi:146 [INFO] found job 17148090 in provider checkpoint proc_lsst.multi INFO: found job 17148090 in provider checkpoint 2024-03-17 04:08:47 proc_lsst.multi:201 [INFO] cancelling 17148090 on provider checkpoint proc_lsst.multi INFO: cancelling 17148090 on provider checkpoint 2024-03-17 04:08:48 proc_lsst.multi:146 [INFO] found job 17148091 in provider checkpoint proc_lsst.multi INFO: found job 17148091 in provider checkpoint 2024-03-17 04:08:48 proc_lsst.multi:201 [INFO] cancelling 17148091 on provider checkpoint proc_lsst.multi INFO: cancelling 17148091 on provider checkpoint 2024-03-17 04:08:48 proc_lsst.multi:146 [INFO] found job 17148092 in provider checkpoint proc_lsst.multi INFO: found job 17148092 in provider checkpoint 2024-03-17 04:08:48 proc_lsst.multi:201 [INFO] cancelling 17148092 on provider checkpoint proc_lsst.multi INFO: cancelling 17148092 on provider checkpoint 2024-03-17 04:08:48 proc_lsst.multi:146 [INFO] found job 17148093 in provider checkpoint proc_lsst.multi INFO: found job 17148093 in provider checkpoint 2024-03-17 04:08:48 proc_lsst.multi:201 [INFO] cancelling 17148093 on provider checkpoint proc_lsst.multi INFO: cancelling 17148093 on provider checkpoint 2024-03-17 04:08:48 proc_lsst.multi:146 [INFO] found job 17148094 in provider checkpoint proc_lsst.multi INFO: found job 17148094 in provider checkpoint 2024-03-17 04:08:48 proc_lsst.multi:201 [INFO] cancelling 17148094 on provider checkpoint proc_lsst.multi INFO: cancelling 17148094 on provider checkpoint 2024-03-17 04:08:49 proc_lsst.multi:146 [INFO] found job 17148095 in provider checkpoint proc_lsst.multi INFO: found job 17148095 in provider checkpoint 2024-03-17 04:08:49 proc_lsst.multi:201 [INFO] cancelling 17148095 on provider checkpoint proc_lsst.multi INFO: cancelling 17148095 on provider checkpoint 2024-03-17 04:08:49 proc_lsst.multi:146 [INFO] found job 17148096 in provider checkpoint proc_lsst.multi INFO: found job 17148096 in provider checkpoint 2024-03-17 04:08:49 proc_lsst.multi:201 [INFO] cancelling 17148096 on provider checkpoint proc_lsst.multi INFO: cancelling 17148096 on provider checkpoint 2024-03-17 04:08:49 proc_lsst.multi:146 [INFO] found job 17148097 in provider checkpoint proc_lsst.multi INFO: found job 17148097 in provider checkpoint 2024-03-17 04:08:49 proc_lsst.multi:201 [INFO] cancelling 17148097 on provider checkpoint proc_lsst.multi INFO: cancelling 17148097 on provider checkpoint 2024-03-17 04:08:50 proc_lsst.multi:146 [INFO] found job 17148098 in provider checkpoint proc_lsst.multi INFO: found job 17148098 in provider checkpoint 2024-03-17 04:08:50 proc_lsst.multi:201 [INFO] cancelling 17148098 on provider checkpoint proc_lsst.multi INFO: cancelling 17148098 on provider checkpoint 2024-03-17 04:08:50 proc_lsst.multi:146 [INFO] found job 17148099 in provider checkpoint proc_lsst.multi INFO: found job 17148099 in provider checkpoint 2024-03-17 04:08:50 proc_lsst.multi:201 [INFO] cancelling 17148099 on provider checkpoint proc_lsst.multi INFO: cancelling 17148099 on provider checkpoint 2024-03-17 04:08:50 proc_lsst.multi:146 [INFO] found job 17148100 in provider checkpoint proc_lsst.multi INFO: found job 17148100 in provider checkpoint 2024-03-17 04:08:50 proc_lsst.multi:201 [INFO] cancelling 17148100 on provider checkpoint proc_lsst.multi INFO: cancelling 17148100 on provider checkpoint 2024-03-17 04:08:50 proc_lsst.multi:146 [INFO] found job 17148101 in provider checkpoint proc_lsst.multi INFO: found job 17148101 in provider checkpoint 2024-03-17 04:08:50 proc_lsst.multi:201 [INFO] cancelling 17148101 on provider checkpoint proc_lsst.multi INFO: cancelling 17148101 on provider checkpoint 2024-03-17 04:08:51 proc_lsst.multi:146 [INFO] found job 17148102 in provider checkpoint proc_lsst.multi INFO: found job 17148102 in provider checkpoint 2024-03-17 04:08:51 proc_lsst.multi:201 [INFO] cancelling 17148102 on provider checkpoint proc_lsst.multi INFO: cancelling 17148102 on provider checkpoint 2024-03-17 04:08:51 proc_lsst.multi:146 [INFO] found job 17148103 in provider checkpoint proc_lsst.multi INFO: found job 17148103 in provider checkpoint 2024-03-17 04:08:51 proc_lsst.multi:201 [INFO] cancelling 17148103 on provider checkpoint proc_lsst.multi INFO: cancelling 17148103 on provider checkpoint 2024-03-17 04:08:51 proc_lsst.multi:146 [INFO] found job 17148104 in provider checkpoint proc_lsst.multi INFO: found job 17148104 in provider checkpoint 2024-03-17 04:08:51 proc_lsst.multi:201 [INFO] cancelling 17148104 on provider checkpoint proc_lsst.multi INFO: cancelling 17148104 on provider checkpoint 2024-03-17 04:08:52 proc_lsst.multi:146 [INFO] found job 17148105 in provider checkpoint proc_lsst.multi INFO: found job 17148105 in provider checkpoint 2024-03-17 04:08:52 proc_lsst.multi:201 [INFO] cancelling 17148105 on provider checkpoint proc_lsst.multi INFO: cancelling 17148105 on provider checkpoint 2024-03-17 04:08:52 proc_lsst.multi:146 [INFO] found job 17148106 in provider checkpoint proc_lsst.multi INFO: found job 17148106 in provider checkpoint 2024-03-17 04:08:52 proc_lsst.multi:201 [INFO] cancelling 17148106 on provider checkpoint proc_lsst.multi INFO: cancelling 17148106 on provider checkpoint 2024-03-17 04:08:52 proc_lsst.multi:146 [INFO] found job 17148107 in provider checkpoint proc_lsst.multi INFO: found job 17148107 in provider checkpoint 2024-03-17 04:08:52 proc_lsst.multi:201 [INFO] cancelling 17148107 on provider checkpoint proc_lsst.multi INFO: cancelling 17148107 on provider checkpoint 2024-03-17 04:08:52 proc_lsst.multi:146 [INFO] found job 17148108 in provider checkpoint proc_lsst.multi INFO: found job 17148108 in provider checkpoint 2024-03-17 04:08:52 proc_lsst.multi:201 [INFO] cancelling 17148108 on provider checkpoint proc_lsst.multi INFO: cancelling 17148108 on provider checkpoint 2024-03-17 04:08:53 proc_lsst.multi:146 [INFO] found job 17148109 in provider checkpoint proc_lsst.multi INFO: found job 17148109 in provider checkpoint 2024-03-17 04:08:53 proc_lsst.multi:201 [INFO] cancelling 17148109 on provider checkpoint proc_lsst.multi INFO: cancelling 17148109 on provider checkpoint 2024-03-17 04:08:53 proc_lsst.multi:146 [INFO] found job 17148110 in provider checkpoint proc_lsst.multi INFO: found job 17148110 in provider checkpoint 2024-03-17 04:08:53 proc_lsst.multi:201 [INFO] cancelling 17148110 on provider checkpoint proc_lsst.multi INFO: cancelling 17148110 on provider checkpoint 2024-03-17 04:08:53 proc_lsst.multi:146 [INFO] found job 17148111 in provider checkpoint proc_lsst.multi INFO: found job 17148111 in provider checkpoint 2024-03-17 04:08:53 proc_lsst.multi:201 [INFO] cancelling 17148111 on provider checkpoint proc_lsst.multi INFO: cancelling 17148111 on provider checkpoint 2024-03-17 04:08:54 proc_lsst.multi:146 [INFO] found job 17148112 in provider checkpoint proc_lsst.multi INFO: found job 17148112 in provider checkpoint 2024-03-17 04:08:54 proc_lsst.multi:201 [INFO] cancelling 17148112 on provider checkpoint proc_lsst.multi INFO: cancelling 17148112 on provider checkpoint 2024-03-17 04:08:54 proc_lsst.multi:146 [INFO] found job 17148113 in provider checkpoint proc_lsst.multi INFO: found job 17148113 in provider checkpoint 2024-03-17 04:08:54 proc_lsst.multi:201 [INFO] cancelling 17148113 on provider checkpoint proc_lsst.multi INFO: cancelling 17148113 on provider checkpoint 2024-03-17 04:08:54 proc_lsst.multi:146 [INFO] found job 17148114 in provider checkpoint proc_lsst.multi INFO: found job 17148114 in provider checkpoint 2024-03-17 04:08:54 proc_lsst.multi:201 [INFO] cancelling 17148114 on provider checkpoint proc_lsst.multi INFO: cancelling 17148114 on provider checkpoint 2024-03-17 04:08:55 proc_lsst.multi:146 [INFO] found job 17148115 in provider checkpoint proc_lsst.multi INFO: found job 17148115 in provider checkpoint 2024-03-17 04:08:55 proc_lsst.multi:201 [INFO] cancelling 17148115 on provider checkpoint proc_lsst.multi INFO: cancelling 17148115 on provider checkpoint 2024-03-17 04:08:55 proc_lsst.multi:146 [INFO] found job 17148116 in provider checkpoint proc_lsst.multi INFO: found job 17148116 in provider checkpoint 2024-03-17 04:08:55 proc_lsst.multi:201 [INFO] cancelling 17148116 on provider checkpoint proc_lsst.multi INFO: cancelling 17148116 on provider checkpoint 2024-03-17 04:08:55 proc_lsst.multi:146 [INFO] found job 17148117 in provider checkpoint proc_lsst.multi INFO: found job 17148117 in provider checkpoint 2024-03-17 04:08:55 proc_lsst.multi:201 [INFO] cancelling 17148117 on provider checkpoint proc_lsst.multi INFO: cancelling 17148117 on provider checkpoint 2024-03-17 04:08:55 proc_lsst.multi:146 [INFO] found job 17148118 in provider checkpoint proc_lsst.multi INFO: found job 17148118 in provider checkpoint 2024-03-17 04:08:55 proc_lsst.multi:201 [INFO] cancelling 17148118 on provider checkpoint proc_lsst.multi INFO: cancelling 17148118 on provider checkpoint 2024-03-17 04:08:56 proc_lsst.multi:146 [INFO] found job 17148119 in provider checkpoint proc_lsst.multi INFO: found job 17148119 in provider checkpoint 2024-03-17 04:08:56 proc_lsst.multi:201 [INFO] cancelling 17148119 on provider checkpoint proc_lsst.multi INFO: cancelling 17148119 on provider checkpoint 2024-03-17 04:08:56 proc_lsst.multi:146 [INFO] found job 17148120 in provider checkpoint proc_lsst.multi INFO: found job 17148120 in provider checkpoint 2024-03-17 04:08:56 proc_lsst.multi:201 [INFO] cancelling 17148120 on provider checkpoint proc_lsst.multi INFO: cancelling 17148120 on provider checkpoint 2024-03-17 04:08:56 proc_lsst.multi:146 [INFO] found job 17148121 in provider checkpoint proc_lsst.multi INFO: found job 17148121 in provider checkpoint 2024-03-17 04:08:56 proc_lsst.multi:201 [INFO] cancelling 17148121 on provider checkpoint proc_lsst.multi INFO: cancelling 17148121 on provider checkpoint 2024-03-17 04:08:57 proc_lsst.multi:146 [INFO] found job 17148122 in provider checkpoint proc_lsst.multi INFO: found job 17148122 in provider checkpoint 2024-03-17 04:08:57 proc_lsst.multi:201 [INFO] cancelling 17148122 on provider checkpoint proc_lsst.multi INFO: cancelling 17148122 on provider checkpoint 2024-03-17 04:08:57 proc_lsst.multi:146 [INFO] found job 17148123 in provider checkpoint proc_lsst.multi INFO: found job 17148123 in provider checkpoint 2024-03-17 04:08:57 proc_lsst.multi:201 [INFO] cancelling 17148123 on provider checkpoint proc_lsst.multi INFO: cancelling 17148123 on provider checkpoint 2024-03-17 04:08:57 proc_lsst.multi:146 [INFO] found job 17148124 in provider checkpoint proc_lsst.multi INFO: found job 17148124 in provider checkpoint 2024-03-17 04:08:57 proc_lsst.multi:201 [INFO] cancelling 17148124 on provider checkpoint proc_lsst.multi INFO: cancelling 17148124 on provider checkpoint 2024-03-17 04:08:57 proc_lsst.multi:146 [INFO] found job 17148125 in provider checkpoint proc_lsst.multi INFO: found job 17148125 in provider checkpoint 2024-03-17 04:08:57 proc_lsst.multi:201 [INFO] cancelling 17148125 on provider checkpoint proc_lsst.multi INFO: cancelling 17148125 on provider checkpoint 2024-03-17 04:08:58 proc_lsst.multi:146 [INFO] found job 17148126 in provider checkpoint proc_lsst.multi INFO: found job 17148126 in provider checkpoint 2024-03-17 04:08:58 proc_lsst.multi:201 [INFO] cancelling 17148126 on provider checkpoint proc_lsst.multi INFO: cancelling 17148126 on provider checkpoint 2024-03-17 04:08:58 proc_lsst.multi:146 [INFO] found job 17148127 in provider checkpoint proc_lsst.multi INFO: found job 17148127 in provider checkpoint 2024-03-17 04:08:58 proc_lsst.multi:201 [INFO] cancelling 17148127 on provider checkpoint proc_lsst.multi INFO: cancelling 17148127 on provider checkpoint 2024-03-17 04:08:58 proc_lsst.multi:146 [INFO] found job 17148128 in provider checkpoint proc_lsst.multi INFO: found job 17148128 in provider checkpoint 2024-03-17 04:08:58 proc_lsst.multi:201 [INFO] cancelling 17148128 on provider checkpoint proc_lsst.multi INFO: cancelling 17148128 on provider checkpoint 2024-03-17 04:08:59 proc_lsst.multi:146 [INFO] found job 17148129 in provider checkpoint proc_lsst.multi INFO: found job 17148129 in provider checkpoint 2024-03-17 04:08:59 proc_lsst.multi:201 [INFO] cancelling 17148129 on provider checkpoint proc_lsst.multi INFO: cancelling 17148129 on provider checkpoint 2024-03-17 04:08:59 proc_lsst.multi:146 [INFO] found job 17148130 in provider checkpoint proc_lsst.multi INFO: found job 17148130 in provider checkpoint 2024-03-17 04:08:59 proc_lsst.multi:201 [INFO] cancelling 17148130 on provider checkpoint proc_lsst.multi INFO: cancelling 17148130 on provider checkpoint 2024-03-17 04:08:59 proc_lsst.multi:146 [INFO] found job 17148131 in provider checkpoint proc_lsst.multi INFO: found job 17148131 in provider checkpoint 2024-03-17 04:08:59 proc_lsst.multi:201 [INFO] cancelling 17148131 on provider checkpoint proc_lsst.multi INFO: cancelling 17148131 on provider checkpoint 2024-03-17 04:09:00 proc_lsst.multi:146 [INFO] found job 17148132 in provider checkpoint proc_lsst.multi INFO: found job 17148132 in provider checkpoint 2024-03-17 04:09:00 proc_lsst.multi:201 [INFO] cancelling 17148132 on provider checkpoint proc_lsst.multi INFO: cancelling 17148132 on provider checkpoint 2024-03-17 04:09:00 proc_lsst.multi:146 [INFO] found job 17148133 in provider checkpoint proc_lsst.multi INFO: found job 17148133 in provider checkpoint 2024-03-17 04:09:00 proc_lsst.multi:201 [INFO] cancelling 17148133 on provider checkpoint proc_lsst.multi INFO: cancelling 17148133 on provider checkpoint 2024-03-17 04:09:00 proc_lsst.multi:146 [INFO] found job 17148134 in provider checkpoint proc_lsst.multi INFO: found job 17148134 in provider checkpoint 2024-03-17 04:09:00 proc_lsst.multi:201 [INFO] cancelling 17148134 on provider checkpoint proc_lsst.multi INFO: cancelling 17148134 on provider checkpoint 2024-03-17 04:09:00 proc_lsst.multi:146 [INFO] found job 17148135 in provider checkpoint proc_lsst.multi INFO: found job 17148135 in provider checkpoint 2024-03-17 04:09:00 proc_lsst.multi:201 [INFO] cancelling 17148135 on provider checkpoint proc_lsst.multi INFO: cancelling 17148135 on provider checkpoint 2024-03-17 04:09:01 proc_lsst.multi:146 [INFO] found job 17148136 in provider checkpoint proc_lsst.multi INFO: found job 17148136 in provider checkpoint 2024-03-17 04:09:01 proc_lsst.multi:201 [INFO] cancelling 17148136 on provider checkpoint proc_lsst.multi INFO: cancelling 17148136 on provider checkpoint 2024-03-17 04:09:02 proc_lsst.multi:146 [INFO] found job 17148137 in provider checkpoint proc_lsst.multi INFO: found job 17148137 in provider checkpoint 2024-03-17 04:09:02 proc_lsst.multi:201 [INFO] cancelling 17148137 on provider checkpoint proc_lsst.multi INFO: cancelling 17148137 on provider checkpoint 2024-03-17 04:09:02 proc_lsst.multi:146 [INFO] found job 17148138 in provider checkpoint proc_lsst.multi INFO: found job 17148138 in provider checkpoint 2024-03-17 04:09:02 proc_lsst.multi:201 [INFO] cancelling 17148138 on provider checkpoint proc_lsst.multi INFO: cancelling 17148138 on provider checkpoint 2024-03-17 04:09:02 proc_lsst.multi:146 [INFO] found job 17148139 in provider checkpoint proc_lsst.multi INFO: found job 17148139 in provider checkpoint 2024-03-17 04:09:02 proc_lsst.multi:201 [INFO] cancelling 17148139 on provider checkpoint proc_lsst.multi INFO: cancelling 17148139 on provider checkpoint 2024-03-17 04:09:02 proc_lsst.multi:146 [INFO] found job 17148140 in provider checkpoint proc_lsst.multi INFO: found job 17148140 in provider checkpoint 2024-03-17 04:09:02 proc_lsst.multi:201 [INFO] cancelling 17148140 on provider checkpoint proc_lsst.multi INFO: cancelling 17148140 on provider checkpoint 2024-03-17 04:09:03 proc_lsst.multi:146 [INFO] found job 17148141 in provider checkpoint proc_lsst.multi INFO: found job 17148141 in provider checkpoint 2024-03-17 04:09:03 proc_lsst.multi:201 [INFO] cancelling 17148141 on provider checkpoint proc_lsst.multi INFO: cancelling 17148141 on provider checkpoint 2024-03-17 04:09:03 proc_lsst.multi:146 [INFO] found job 17148142 in provider checkpoint proc_lsst.multi INFO: found job 17148142 in provider checkpoint 2024-03-17 04:09:03 proc_lsst.multi:201 [INFO] cancelling 17148142 on provider checkpoint proc_lsst.multi INFO: cancelling 17148142 on provider checkpoint 2024-03-17 04:09:03 proc_lsst.multi:146 [INFO] found job 17148143 in provider checkpoint proc_lsst.multi INFO: found job 17148143 in provider checkpoint 2024-03-17 04:09:03 proc_lsst.multi:201 [INFO] cancelling 17148143 on provider checkpoint proc_lsst.multi INFO: cancelling 17148143 on provider checkpoint 2024-03-17 04:09:04 proc_lsst.multi:146 [INFO] found job 17148144 in provider checkpoint proc_lsst.multi INFO: found job 17148144 in provider checkpoint 2024-03-17 04:09:04 proc_lsst.multi:201 [INFO] cancelling 17148144 on provider checkpoint proc_lsst.multi INFO: cancelling 17148144 on provider checkpoint 2024-03-17 04:09:04 proc_lsst.multi:146 [INFO] found job 17148145 in provider checkpoint proc_lsst.multi INFO: found job 17148145 in provider checkpoint 2024-03-17 04:09:04 proc_lsst.multi:201 [INFO] cancelling 17148145 on provider checkpoint proc_lsst.multi INFO: cancelling 17148145 on provider checkpoint 2024-03-17 04:09:04 proc_lsst.multi:146 [INFO] found job 17148146 in provider checkpoint proc_lsst.multi INFO: found job 17148146 in provider checkpoint 2024-03-17 04:09:04 proc_lsst.multi:201 [INFO] cancelling 17148146 on provider checkpoint proc_lsst.multi INFO: cancelling 17148146 on provider checkpoint 2024-03-17 04:09:05 proc_lsst.multi:146 [INFO] found job 17148147 in provider checkpoint proc_lsst.multi INFO: found job 17148147 in provider checkpoint 2024-03-17 04:09:05 proc_lsst.multi:201 [INFO] cancelling 17148147 on provider checkpoint proc_lsst.multi INFO: cancelling 17148147 on provider checkpoint 2024-03-17 04:09:05 proc_lsst.multi:146 [INFO] found job 17148148 in provider checkpoint proc_lsst.multi INFO: found job 17148148 in provider checkpoint 2024-03-17 04:09:05 proc_lsst.multi:201 [INFO] cancelling 17148148 on provider checkpoint proc_lsst.multi INFO: cancelling 17148148 on provider checkpoint 2024-03-17 04:09:05 proc_lsst.multi:146 [INFO] found job 17148149 in provider checkpoint proc_lsst.multi INFO: found job 17148149 in provider checkpoint 2024-03-17 04:09:05 proc_lsst.multi:201 [INFO] cancelling 17148149 on provider checkpoint proc_lsst.multi INFO: cancelling 17148149 on provider checkpoint 2024-03-17 04:09:06 proc_lsst.multi:146 [INFO] found job 17148150 in provider checkpoint proc_lsst.multi INFO: found job 17148150 in provider checkpoint 2024-03-17 04:09:06 proc_lsst.multi:201 [INFO] cancelling 17148150 on provider checkpoint proc_lsst.multi INFO: cancelling 17148150 on provider checkpoint 2024-03-17 04:09:06 proc_lsst.multi:146 [INFO] found job 17148151 in provider checkpoint proc_lsst.multi INFO: found job 17148151 in provider checkpoint 2024-03-17 04:09:06 proc_lsst.multi:201 [INFO] cancelling 17148151 on provider checkpoint proc_lsst.multi INFO: cancelling 17148151 on provider checkpoint 2024-03-17 04:09:06 proc_lsst.multi:146 [INFO] found job 17148152 in provider checkpoint proc_lsst.multi INFO: found job 17148152 in provider checkpoint 2024-03-17 04:09:06 proc_lsst.multi:201 [INFO] cancelling 17148152 on provider checkpoint proc_lsst.multi INFO: cancelling 17148152 on provider checkpoint 2024-03-17 04:09:07 proc_lsst.multi:146 [INFO] found job 17148153 in provider checkpoint proc_lsst.multi INFO: found job 17148153 in provider checkpoint 2024-03-17 04:09:07 proc_lsst.multi:201 [INFO] cancelling 17148153 on provider checkpoint proc_lsst.multi INFO: cancelling 17148153 on provider checkpoint 2024-03-17 04:09:07 proc_lsst.multi:146 [INFO] found job 17148154 in provider checkpoint proc_lsst.multi INFO: found job 17148154 in provider checkpoint 2024-03-17 04:09:07 proc_lsst.multi:201 [INFO] cancelling 17148154 on provider checkpoint proc_lsst.multi INFO: cancelling 17148154 on provider checkpoint 2024-03-17 04:09:07 proc_lsst.multi:146 [INFO] found job 17148155 in provider checkpoint proc_lsst.multi INFO: found job 17148155 in provider checkpoint 2024-03-17 04:09:07 proc_lsst.multi:201 [INFO] cancelling 17148155 on provider checkpoint proc_lsst.multi INFO: cancelling 17148155 on provider checkpoint 2024-03-17 04:09:07 proc_lsst.multi:146 [INFO] found job 17148156 in provider checkpoint proc_lsst.multi INFO: found job 17148156 in provider checkpoint 2024-03-17 04:09:07 proc_lsst.multi:201 [INFO] cancelling 17148156 on provider checkpoint proc_lsst.multi INFO: cancelling 17148156 on provider checkpoint 2024-03-17 04:09:08 proc_lsst.multi:146 [INFO] found job 17148157 in provider checkpoint proc_lsst.multi INFO: found job 17148157 in provider checkpoint 2024-03-17 04:09:08 proc_lsst.multi:201 [INFO] cancelling 17148157 on provider checkpoint proc_lsst.multi INFO: cancelling 17148157 on provider checkpoint 2024-03-17 04:09:08 proc_lsst.multi:146 [INFO] found job 17148158 in provider checkpoint proc_lsst.multi INFO: found job 17148158 in provider checkpoint 2024-03-17 04:09:08 proc_lsst.multi:201 [INFO] cancelling 17148158 on provider checkpoint proc_lsst.multi INFO: cancelling 17148158 on provider checkpoint 2024-03-17 04:09:08 proc_lsst.multi:146 [INFO] found job 17148159 in provider checkpoint proc_lsst.multi INFO: found job 17148159 in provider checkpoint 2024-03-17 04:09:08 proc_lsst.multi:201 [INFO] cancelling 17148159 on provider checkpoint proc_lsst.multi INFO: cancelling 17148159 on provider checkpoint 2024-03-17 04:09:09 proc_lsst.multi:146 [INFO] found job 17148160 in provider checkpoint proc_lsst.multi INFO: found job 17148160 in provider checkpoint 2024-03-17 04:09:09 proc_lsst.multi:201 [INFO] cancelling 17148160 on provider checkpoint proc_lsst.multi INFO: cancelling 17148160 on provider checkpoint 2024-03-17 04:09:09 proc_lsst.multi:146 [INFO] found job 17148161 in provider checkpoint proc_lsst.multi INFO: found job 17148161 in provider checkpoint 2024-03-17 04:09:09 proc_lsst.multi:201 [INFO] cancelling 17148161 on provider checkpoint proc_lsst.multi INFO: cancelling 17148161 on provider checkpoint 2024-03-17 04:09:09 proc_lsst.multi:146 [INFO] found job 17148162 in provider checkpoint proc_lsst.multi INFO: found job 17148162 in provider checkpoint 2024-03-17 04:09:09 proc_lsst.multi:201 [INFO] cancelling 17148162 on provider checkpoint proc_lsst.multi INFO: cancelling 17148162 on provider checkpoint 2024-03-17 04:09:10 proc_lsst.multi:146 [INFO] found job 17148163 in provider checkpoint proc_lsst.multi INFO: found job 17148163 in provider checkpoint 2024-03-17 04:09:10 proc_lsst.multi:201 [INFO] cancelling 17148163 on provider checkpoint proc_lsst.multi INFO: cancelling 17148163 on provider checkpoint 2024-03-17 04:09:10 proc_lsst.multi:146 [INFO] found job 17148166 in provider checkpoint proc_lsst.multi INFO: found job 17148166 in provider checkpoint 2024-03-17 04:09:10 proc_lsst.multi:201 [INFO] cancelling 17148166 on provider checkpoint proc_lsst.multi INFO: cancelling 17148166 on provider checkpoint 2024-03-17 04:09:10 proc_lsst.multi:146 [INFO] found job 17148167 in provider checkpoint proc_lsst.multi INFO: found job 17148167 in provider checkpoint 2024-03-17 04:09:10 proc_lsst.multi:201 [INFO] cancelling 17148167 on provider checkpoint proc_lsst.multi INFO: cancelling 17148167 on provider checkpoint 2024-03-17 04:09:11 proc_lsst.multi:146 [INFO] found job 17148168 in provider checkpoint proc_lsst.multi INFO: found job 17148168 in provider checkpoint 2024-03-17 04:09:11 proc_lsst.multi:201 [INFO] cancelling 17148168 on provider checkpoint proc_lsst.multi INFO: cancelling 17148168 on provider checkpoint 2024-03-17 04:09:11 proc_lsst.multi:146 [INFO] found job 17148169 in provider checkpoint proc_lsst.multi INFO: found job 17148169 in provider checkpoint 2024-03-17 04:09:11 proc_lsst.multi:201 [INFO] cancelling 17148169 on provider checkpoint proc_lsst.multi INFO: cancelling 17148169 on provider checkpoint 2024-03-17 04:09:11 proc_lsst.multi:146 [INFO] found job 17148170 in provider checkpoint proc_lsst.multi INFO: found job 17148170 in provider checkpoint 2024-03-17 04:09:11 proc_lsst.multi:201 [INFO] cancelling 17148170 on provider checkpoint proc_lsst.multi INFO: cancelling 17148170 on provider checkpoint 2024-03-17 04:09:11 proc_lsst.multi:146 [INFO] found job 17148171 in provider checkpoint proc_lsst.multi INFO: found job 17148171 in provider checkpoint 2024-03-17 04:09:11 proc_lsst.multi:201 [INFO] cancelling 17148171 on provider checkpoint proc_lsst.multi INFO: cancelling 17148171 on provider checkpoint 2024-03-17 04:09:12 proc_lsst.multi:146 [INFO] found job 17148172 in provider checkpoint proc_lsst.multi INFO: found job 17148172 in provider checkpoint 2024-03-17 04:09:12 proc_lsst.multi:201 [INFO] cancelling 17148172 on provider checkpoint proc_lsst.multi INFO: cancelling 17148172 on provider checkpoint 2024-03-17 04:09:12 proc_lsst.multi:146 [INFO] found job 17148173 in provider checkpoint proc_lsst.multi INFO: found job 17148173 in provider checkpoint 2024-03-17 04:09:12 proc_lsst.multi:201 [INFO] cancelling 17148173 on provider checkpoint proc_lsst.multi INFO: cancelling 17148173 on provider checkpoint 2024-03-17 04:09:12 proc_lsst.multi:146 [INFO] found job 17148174 in provider checkpoint proc_lsst.multi INFO: found job 17148174 in provider checkpoint 2024-03-17 04:09:12 proc_lsst.multi:201 [INFO] cancelling 17148174 on provider checkpoint proc_lsst.multi INFO: cancelling 17148174 on provider checkpoint 2024-03-17 04:09:13 proc_lsst.multi:146 [INFO] found job 17148175 in provider checkpoint proc_lsst.multi INFO: found job 17148175 in provider checkpoint 2024-03-17 04:09:13 proc_lsst.multi:201 [INFO] cancelling 17148175 on provider checkpoint proc_lsst.multi INFO: cancelling 17148175 on provider checkpoint 2024-03-17 04:09:13 proc_lsst.multi:146 [INFO] found job 17148176 in provider checkpoint proc_lsst.multi INFO: found job 17148176 in provider checkpoint 2024-03-17 04:09:13 proc_lsst.multi:201 [INFO] cancelling 17148176 on provider checkpoint proc_lsst.multi INFO: cancelling 17148176 on provider checkpoint 2024-03-17 04:09:13 proc_lsst.multi:146 [INFO] found job 17148177 in provider checkpoint proc_lsst.multi INFO: found job 17148177 in provider checkpoint 2024-03-17 04:09:13 proc_lsst.multi:201 [INFO] cancelling 17148177 on provider checkpoint proc_lsst.multi INFO: cancelling 17148177 on provider checkpoint 2024-03-17 04:09:14 proc_lsst.multi:146 [INFO] found job 17148178 in provider checkpoint proc_lsst.multi INFO: found job 17148178 in provider checkpoint 2024-03-17 04:09:14 proc_lsst.multi:201 [INFO] cancelling 17148178 on provider checkpoint proc_lsst.multi INFO: cancelling 17148178 on provider checkpoint 2024-03-17 04:09:14 proc_lsst.multi:146 [INFO] found job 17148179 in provider checkpoint proc_lsst.multi INFO: found job 17148179 in provider checkpoint 2024-03-17 04:09:14 proc_lsst.multi:201 [INFO] cancelling 17148179 on provider checkpoint proc_lsst.multi INFO: cancelling 17148179 on provider checkpoint 2024-03-17 04:09:14 proc_lsst.multi:146 [INFO] found job 17148180 in provider checkpoint proc_lsst.multi INFO: found job 17148180 in provider checkpoint 2024-03-17 04:09:14 proc_lsst.multi:201 [INFO] cancelling 17148180 on provider checkpoint proc_lsst.multi INFO: cancelling 17148180 on provider checkpoint 2024-03-17 04:09:15 proc_lsst.multi:146 [INFO] found job 17148181 in provider checkpoint proc_lsst.multi INFO: found job 17148181 in provider checkpoint 2024-03-17 04:09:15 proc_lsst.multi:201 [INFO] cancelling 17148181 on provider checkpoint proc_lsst.multi INFO: cancelling 17148181 on provider checkpoint 2024-03-17 04:09:15 proc_lsst.multi:146 [INFO] found job 17148182 in provider checkpoint proc_lsst.multi INFO: found job 17148182 in provider checkpoint 2024-03-17 04:09:15 proc_lsst.multi:201 [INFO] cancelling 17148182 on provider checkpoint proc_lsst.multi INFO: cancelling 17148182 on provider checkpoint 2024-03-17 04:09:15 proc_lsst.multi:146 [INFO] found job 17148183 in provider checkpoint proc_lsst.multi INFO: found job 17148183 in provider checkpoint 2024-03-17 04:09:15 proc_lsst.multi:201 [INFO] cancelling 17148183 on provider checkpoint proc_lsst.multi INFO: cancelling 17148183 on provider checkpoint 2024-03-17 04:09:15 proc_lsst.multi:146 [INFO] found job 17148184 in provider checkpoint proc_lsst.multi INFO: found job 17148184 in provider checkpoint 2024-03-17 04:09:15 proc_lsst.multi:201 [INFO] cancelling 17148184 on provider checkpoint proc_lsst.multi INFO: cancelling 17148184 on provider checkpoint 2024-03-17 04:09:16 proc_lsst.multi:146 [INFO] found job 17148185 in provider checkpoint proc_lsst.multi INFO: found job 17148185 in provider checkpoint 2024-03-17 04:09:16 proc_lsst.multi:201 [INFO] cancelling 17148185 on provider checkpoint proc_lsst.multi INFO: cancelling 17148185 on provider checkpoint 2024-03-17 04:09:16 proc_lsst.multi:146 [INFO] found job 17148186 in provider checkpoint proc_lsst.multi INFO: found job 17148186 in provider checkpoint 2024-03-17 04:09:16 proc_lsst.multi:201 [INFO] cancelling 17148186 on provider checkpoint proc_lsst.multi INFO: cancelling 17148186 on provider checkpoint 2024-03-17 04:09:16 proc_lsst.multi:146 [INFO] found job 17148187 in provider checkpoint proc_lsst.multi INFO: found job 17148187 in provider checkpoint 2024-03-17 04:09:16 proc_lsst.multi:201 [INFO] cancelling 17148187 on provider checkpoint proc_lsst.multi INFO: cancelling 17148187 on provider checkpoint 2024-03-17 04:09:17 proc_lsst.multi:146 [INFO] found job 17148188 in provider checkpoint proc_lsst.multi INFO: found job 17148188 in provider checkpoint 2024-03-17 04:09:17 proc_lsst.multi:201 [INFO] cancelling 17148188 on provider checkpoint proc_lsst.multi INFO: cancelling 17148188 on provider checkpoint 2024-03-17 04:09:17 proc_lsst.multi:146 [INFO] found job 17148189 in provider checkpoint proc_lsst.multi INFO: found job 17148189 in provider checkpoint 2024-03-17 04:09:17 proc_lsst.multi:201 [INFO] cancelling 17148189 on provider checkpoint proc_lsst.multi INFO: cancelling 17148189 on provider checkpoint 2024-03-17 04:09:17 proc_lsst.multi:146 [INFO] found job 17148190 in provider checkpoint proc_lsst.multi INFO: found job 17148190 in provider checkpoint 2024-03-17 04:09:17 proc_lsst.multi:201 [INFO] cancelling 17148190 on provider checkpoint proc_lsst.multi INFO: cancelling 17148190 on provider checkpoint 2024-03-17 04:09:18 proc_lsst.multi:146 [INFO] found job 17148191 in provider checkpoint proc_lsst.multi INFO: found job 17148191 in provider checkpoint 2024-03-17 04:09:18 proc_lsst.multi:201 [INFO] cancelling 17148191 on provider checkpoint proc_lsst.multi INFO: cancelling 17148191 on provider checkpoint 2024-03-17 04:09:18 proc_lsst.multi:146 [INFO] found job 17148192 in provider checkpoint proc_lsst.multi INFO: found job 17148192 in provider checkpoint 2024-03-17 04:09:18 proc_lsst.multi:201 [INFO] cancelling 17148192 on provider checkpoint proc_lsst.multi INFO: cancelling 17148192 on provider checkpoint 2024-03-17 04:09:18 proc_lsst.multi:146 [INFO] found job 17148193 in provider checkpoint proc_lsst.multi INFO: found job 17148193 in provider checkpoint 2024-03-17 04:09:18 proc_lsst.multi:201 [INFO] cancelling 17148193 on provider checkpoint proc_lsst.multi INFO: cancelling 17148193 on provider checkpoint 2024-03-17 04:09:19 proc_lsst.multi:146 [INFO] found job 17148194 in provider checkpoint proc_lsst.multi INFO: found job 17148194 in provider checkpoint 2024-03-17 04:09:19 proc_lsst.multi:201 [INFO] cancelling 17148194 on provider checkpoint proc_lsst.multi INFO: cancelling 17148194 on provider checkpoint 2024-03-17 04:09:19 proc_lsst.multi:146 [INFO] found job 17148195 in provider checkpoint proc_lsst.multi INFO: found job 17148195 in provider checkpoint 2024-03-17 04:09:19 proc_lsst.multi:201 [INFO] cancelling 17148195 on provider checkpoint proc_lsst.multi INFO: cancelling 17148195 on provider checkpoint 2024-03-17 04:09:19 proc_lsst.multi:146 [INFO] found job 17148196 in provider checkpoint proc_lsst.multi INFO: found job 17148196 in provider checkpoint 2024-03-17 04:09:19 proc_lsst.multi:201 [INFO] cancelling 17148196 on provider checkpoint proc_lsst.multi INFO: cancelling 17148196 on provider checkpoint 2024-03-17 04:09:20 proc_lsst.multi:146 [INFO] found job 17148197 in provider checkpoint proc_lsst.multi INFO: found job 17148197 in provider checkpoint 2024-03-17 04:09:20 proc_lsst.multi:201 [INFO] cancelling 17148197 on provider checkpoint proc_lsst.multi INFO: cancelling 17148197 on provider checkpoint 2024-03-17 04:09:20 proc_lsst.multi:146 [INFO] found job 17148198 in provider checkpoint proc_lsst.multi INFO: found job 17148198 in provider checkpoint 2024-03-17 04:09:20 proc_lsst.multi:201 [INFO] cancelling 17148198 on provider checkpoint proc_lsst.multi INFO: cancelling 17148198 on provider checkpoint 2024-03-17 04:09:20 proc_lsst.multi:146 [INFO] found job 17148199 in provider checkpoint proc_lsst.multi INFO: found job 17148199 in provider checkpoint 2024-03-17 04:09:20 proc_lsst.multi:201 [INFO] cancelling 17148199 on provider checkpoint proc_lsst.multi INFO: cancelling 17148199 on provider checkpoint 2024-03-17 04:09:20 proc_lsst.multi:146 [INFO] found job 17148200 in provider checkpoint proc_lsst.multi INFO: found job 17148200 in provider checkpoint 2024-03-17 04:09:20 proc_lsst.multi:201 [INFO] cancelling 17148200 on provider checkpoint proc_lsst.multi INFO: cancelling 17148200 on provider checkpoint 2024-03-17 04:09:21 proc_lsst.multi:146 [INFO] found job 17148201 in provider checkpoint proc_lsst.multi INFO: found job 17148201 in provider checkpoint 2024-03-17 04:09:21 proc_lsst.multi:201 [INFO] cancelling 17148201 on provider checkpoint proc_lsst.multi INFO: cancelling 17148201 on provider checkpoint 2024-03-17 04:09:21 proc_lsst.multi:146 [INFO] found job 17148202 in provider checkpoint proc_lsst.multi INFO: found job 17148202 in provider checkpoint 2024-03-17 04:09:21 proc_lsst.multi:201 [INFO] cancelling 17148202 on provider checkpoint proc_lsst.multi INFO: cancelling 17148202 on provider checkpoint 2024-03-17 04:09:21 proc_lsst.multi:146 [INFO] found job 17148203 in provider checkpoint proc_lsst.multi INFO: found job 17148203 in provider checkpoint 2024-03-17 04:09:21 proc_lsst.multi:201 [INFO] cancelling 17148203 on provider checkpoint proc_lsst.multi INFO: cancelling 17148203 on provider checkpoint 2024-03-17 04:09:22 proc_lsst.multi:146 [INFO] found job 17148204 in provider checkpoint proc_lsst.multi INFO: found job 17148204 in provider checkpoint 2024-03-17 04:09:22 proc_lsst.multi:201 [INFO] cancelling 17148204 on provider checkpoint proc_lsst.multi INFO: cancelling 17148204 on provider checkpoint 2024-03-17 04:09:22 proc_lsst.multi:146 [INFO] found job 17148205 in provider checkpoint proc_lsst.multi INFO: found job 17148205 in provider checkpoint 2024-03-17 04:09:22 proc_lsst.multi:201 [INFO] cancelling 17148205 on provider checkpoint proc_lsst.multi INFO: cancelling 17148205 on provider checkpoint 2024-03-17 04:09:22 proc_lsst.multi:146 [INFO] found job 17148206 in provider checkpoint proc_lsst.multi INFO: found job 17148206 in provider checkpoint 2024-03-17 04:09:22 proc_lsst.multi:201 [INFO] cancelling 17148206 on provider checkpoint proc_lsst.multi INFO: cancelling 17148206 on provider checkpoint 2024-03-17 04:09:23 proc_lsst.multi:146 [INFO] found job 17148207 in provider checkpoint proc_lsst.multi INFO: found job 17148207 in provider checkpoint 2024-03-17 04:09:23 proc_lsst.multi:201 [INFO] cancelling 17148207 on provider checkpoint proc_lsst.multi INFO: cancelling 17148207 on provider checkpoint 2024-03-17 04:09:23 proc_lsst.multi:146 [INFO] found job 17148208 in provider checkpoint proc_lsst.multi INFO: found job 17148208 in provider checkpoint 2024-03-17 04:09:23 proc_lsst.multi:201 [INFO] cancelling 17148208 on provider checkpoint proc_lsst.multi INFO: cancelling 17148208 on provider checkpoint 2024-03-17 04:09:23 proc_lsst.multi:146 [INFO] found job 17148209 in provider checkpoint proc_lsst.multi INFO: found job 17148209 in provider checkpoint 2024-03-17 04:09:23 proc_lsst.multi:201 [INFO] cancelling 17148209 on provider checkpoint proc_lsst.multi INFO: cancelling 17148209 on provider checkpoint 2024-03-17 04:09:24 proc_lsst.multi:146 [INFO] found job 17148210 in provider checkpoint proc_lsst.multi INFO: found job 17148210 in provider checkpoint 2024-03-17 04:09:24 proc_lsst.multi:201 [INFO] cancelling 17148210 on provider checkpoint proc_lsst.multi INFO: cancelling 17148210 on provider checkpoint 2024-03-17 04:09:24 proc_lsst.multi:146 [INFO] found job 17148211 in provider checkpoint proc_lsst.multi INFO: found job 17148211 in provider checkpoint 2024-03-17 04:09:24 proc_lsst.multi:201 [INFO] cancelling 17148211 on provider checkpoint proc_lsst.multi INFO: cancelling 17148211 on provider checkpoint 2024-03-17 04:09:24 proc_lsst.multi:146 [INFO] found job 17148212 in provider checkpoint proc_lsst.multi INFO: found job 17148212 in provider checkpoint 2024-03-17 04:09:24 proc_lsst.multi:201 [INFO] cancelling 17148212 on provider checkpoint proc_lsst.multi INFO: cancelling 17148212 on provider checkpoint 2024-03-17 04:09:24 proc_lsst.multi:146 [INFO] found job 17148213 in provider checkpoint proc_lsst.multi INFO: found job 17148213 in provider checkpoint 2024-03-17 04:09:24 proc_lsst.multi:201 [INFO] cancelling 17148213 on provider checkpoint proc_lsst.multi INFO: cancelling 17148213 on provider checkpoint 2024-03-17 04:09:25 proc_lsst.multi:146 [INFO] found job 17148214 in provider checkpoint proc_lsst.multi INFO: found job 17148214 in provider checkpoint 2024-03-17 04:09:25 proc_lsst.multi:201 [INFO] cancelling 17148214 on provider checkpoint proc_lsst.multi INFO: cancelling 17148214 on provider checkpoint 2024-03-17 04:09:25 proc_lsst.multi:146 [INFO] found job 17148215 in provider checkpoint proc_lsst.multi INFO: found job 17148215 in provider checkpoint 2024-03-17 04:09:25 proc_lsst.multi:201 [INFO] cancelling 17148215 on provider checkpoint proc_lsst.multi INFO: cancelling 17148215 on provider checkpoint 2024-03-17 04:09:25 proc_lsst.multi:146 [INFO] found job 17148216 in provider checkpoint proc_lsst.multi INFO: found job 17148216 in provider checkpoint 2024-03-17 04:09:25 proc_lsst.multi:201 [INFO] cancelling 17148216 on provider checkpoint proc_lsst.multi INFO: cancelling 17148216 on provider checkpoint 2024-03-17 04:09:26 proc_lsst.multi:146 [INFO] found job 17148217 in provider checkpoint proc_lsst.multi INFO: found job 17148217 in provider checkpoint 2024-03-17 04:09:26 proc_lsst.multi:201 [INFO] cancelling 17148217 on provider checkpoint proc_lsst.multi INFO: cancelling 17148217 on provider checkpoint 2024-03-17 04:09:26 proc_lsst.multi:146 [INFO] found job 17148218 in provider checkpoint proc_lsst.multi INFO: found job 17148218 in provider checkpoint 2024-03-17 04:09:26 proc_lsst.multi:201 [INFO] cancelling 17148218 on provider checkpoint proc_lsst.multi INFO: cancelling 17148218 on provider checkpoint 2024-03-17 04:09:26 proc_lsst.multi:146 [INFO] found job 17148219 in provider checkpoint proc_lsst.multi INFO: found job 17148219 in provider checkpoint 2024-03-17 04:09:26 proc_lsst.multi:201 [INFO] cancelling 17148219 on provider checkpoint proc_lsst.multi INFO: cancelling 17148219 on provider checkpoint 2024-03-17 04:09:27 proc_lsst.multi:146 [INFO] found job 17148220 in provider checkpoint proc_lsst.multi INFO: found job 17148220 in provider checkpoint 2024-03-17 04:09:27 proc_lsst.multi:201 [INFO] cancelling 17148220 on provider checkpoint proc_lsst.multi INFO: cancelling 17148220 on provider checkpoint 2024-03-17 04:09:27 proc_lsst.multi:146 [INFO] found job 17148221 in provider checkpoint proc_lsst.multi INFO: found job 17148221 in provider checkpoint 2024-03-17 04:09:27 proc_lsst.multi:201 [INFO] cancelling 17148221 on provider checkpoint proc_lsst.multi INFO: cancelling 17148221 on provider checkpoint 2024-03-17 04:09:27 proc_lsst.multi:146 [INFO] found job 17148222 in provider checkpoint proc_lsst.multi INFO: found job 17148222 in provider checkpoint 2024-03-17 04:09:27 proc_lsst.multi:201 [INFO] cancelling 17148222 on provider checkpoint proc_lsst.multi INFO: cancelling 17148222 on provider checkpoint 2024-03-17 04:09:28 proc_lsst.multi:146 [INFO] found job 17148223 in provider checkpoint proc_lsst.multi INFO: found job 17148223 in provider checkpoint 2024-03-17 04:09:28 proc_lsst.multi:201 [INFO] cancelling 17148223 on provider checkpoint proc_lsst.multi INFO: cancelling 17148223 on provider checkpoint 2024-03-17 04:09:28 proc_lsst.multi:146 [INFO] found job 17148224 in provider checkpoint proc_lsst.multi INFO: found job 17148224 in provider checkpoint 2024-03-17 04:09:28 proc_lsst.multi:201 [INFO] cancelling 17148224 on provider checkpoint proc_lsst.multi INFO: cancelling 17148224 on provider checkpoint 2024-03-17 04:09:28 proc_lsst.multi:146 [INFO] found job 17148225 in provider checkpoint proc_lsst.multi INFO: found job 17148225 in provider checkpoint 2024-03-17 04:09:28 proc_lsst.multi:201 [INFO] cancelling 17148225 on provider checkpoint proc_lsst.multi INFO: cancelling 17148225 on provider checkpoint 2024-03-17 04:09:28 proc_lsst.multi:146 [INFO] found job 17148226 in provider checkpoint proc_lsst.multi INFO: found job 17148226 in provider checkpoint 2024-03-17 04:09:28 proc_lsst.multi:201 [INFO] cancelling 17148226 on provider checkpoint proc_lsst.multi INFO: cancelling 17148226 on provider checkpoint 2024-03-17 04:09:29 proc_lsst.multi:146 [INFO] found job 17148227 in provider checkpoint proc_lsst.multi INFO: found job 17148227 in provider checkpoint 2024-03-17 04:09:29 proc_lsst.multi:201 [INFO] cancelling 17148227 on provider checkpoint proc_lsst.multi INFO: cancelling 17148227 on provider checkpoint 2024-03-17 04:09:29 proc_lsst.multi:146 [INFO] found job 17148228 in provider checkpoint proc_lsst.multi INFO: found job 17148228 in provider checkpoint 2024-03-17 04:09:29 proc_lsst.multi:201 [INFO] cancelling 17148228 on provider checkpoint proc_lsst.multi INFO: cancelling 17148228 on provider checkpoint 2024-03-17 04:09:29 proc_lsst.multi:146 [INFO] found job 17148229 in provider checkpoint proc_lsst.multi INFO: found job 17148229 in provider checkpoint 2024-03-17 04:09:29 proc_lsst.multi:201 [INFO] cancelling 17148229 on provider checkpoint proc_lsst.multi INFO: cancelling 17148229 on provider checkpoint 2024-03-17 04:09:30 proc_lsst.multi:146 [INFO] found job 17148230 in provider checkpoint proc_lsst.multi INFO: found job 17148230 in provider checkpoint 2024-03-17 04:09:30 proc_lsst.multi:201 [INFO] cancelling 17148230 on provider checkpoint proc_lsst.multi INFO: cancelling 17148230 on provider checkpoint 2024-03-17 04:09:30 proc_lsst.multi:146 [INFO] found job 17148231 in provider checkpoint proc_lsst.multi INFO: found job 17148231 in provider checkpoint 2024-03-17 04:09:30 proc_lsst.multi:201 [INFO] cancelling 17148231 on provider checkpoint proc_lsst.multi INFO: cancelling 17148231 on provider checkpoint 2024-03-17 04:09:30 proc_lsst.multi:146 [INFO] found job 17148232 in provider checkpoint proc_lsst.multi INFO: found job 17148232 in provider checkpoint 2024-03-17 04:09:30 proc_lsst.multi:201 [INFO] cancelling 17148232 on provider checkpoint proc_lsst.multi INFO: cancelling 17148232 on provider checkpoint 2024-03-17 04:09:30 proc_lsst.multi:146 [INFO] found job 17148233 in provider checkpoint proc_lsst.multi INFO: found job 17148233 in provider checkpoint 2024-03-17 04:09:30 proc_lsst.multi:201 [INFO] cancelling 17148233 on provider checkpoint proc_lsst.multi INFO: cancelling 17148233 on provider checkpoint 2024-03-17 04:09:31 proc_lsst.multi:146 [INFO] found job 17148234 in provider checkpoint proc_lsst.multi INFO: found job 17148234 in provider checkpoint 2024-03-17 04:09:31 proc_lsst.multi:201 [INFO] cancelling 17148234 on provider checkpoint proc_lsst.multi INFO: cancelling 17148234 on provider checkpoint 2024-03-17 04:09:31 proc_lsst.multi:146 [INFO] found job 17148235 in provider checkpoint proc_lsst.multi INFO: found job 17148235 in provider checkpoint 2024-03-17 04:09:31 proc_lsst.multi:201 [INFO] cancelling 17148235 on provider checkpoint proc_lsst.multi INFO: cancelling 17148235 on provider checkpoint 2024-03-17 04:09:31 proc_lsst.multi:146 [INFO] found job 17148236 in provider checkpoint proc_lsst.multi INFO: found job 17148236 in provider checkpoint 2024-03-17 04:09:31 proc_lsst.multi:201 [INFO] cancelling 17148236 on provider checkpoint proc_lsst.multi INFO: cancelling 17148236 on provider checkpoint 2024-03-17 04:09:33 proc_lsst.multi:146 [INFO] found job 17148237 in provider checkpoint proc_lsst.multi INFO: found job 17148237 in provider checkpoint 2024-03-17 04:09:33 proc_lsst.multi:201 [INFO] cancelling 17148237 on provider checkpoint proc_lsst.multi INFO: cancelling 17148237 on provider checkpoint 2024-03-17 04:09:33 proc_lsst.multi:146 [INFO] found job 17148238 in provider checkpoint proc_lsst.multi INFO: found job 17148238 in provider checkpoint 2024-03-17 04:09:33 proc_lsst.multi:201 [INFO] cancelling 17148238 on provider checkpoint proc_lsst.multi INFO: cancelling 17148238 on provider checkpoint 2024-03-17 04:09:33 proc_lsst.multi:146 [INFO] found job 17148239 in provider checkpoint proc_lsst.multi INFO: found job 17148239 in provider checkpoint 2024-03-17 04:09:33 proc_lsst.multi:201 [INFO] cancelling 17148239 on provider checkpoint proc_lsst.multi INFO: cancelling 17148239 on provider checkpoint 2024-03-17 04:09:33 proc_lsst.multi:146 [INFO] found job 17148240 in provider checkpoint proc_lsst.multi INFO: found job 17148240 in provider checkpoint 2024-03-17 04:09:33 proc_lsst.multi:201 [INFO] cancelling 17148240 on provider checkpoint proc_lsst.multi INFO: cancelling 17148240 on provider checkpoint 2024-03-17 04:09:34 proc_lsst.multi:146 [INFO] found job 17148241 in provider checkpoint proc_lsst.multi INFO: found job 17148241 in provider checkpoint 2024-03-17 04:09:34 proc_lsst.multi:201 [INFO] cancelling 17148241 on provider checkpoint proc_lsst.multi INFO: cancelling 17148241 on provider checkpoint 2024-03-17 04:09:34 proc_lsst.multi:146 [INFO] found job 17148242 in provider checkpoint proc_lsst.multi INFO: found job 17148242 in provider checkpoint 2024-03-17 04:09:34 proc_lsst.multi:201 [INFO] cancelling 17148242 on provider checkpoint proc_lsst.multi INFO: cancelling 17148242 on provider checkpoint 2024-03-17 04:09:34 proc_lsst.multi:146 [INFO] found job 17148243 in provider checkpoint proc_lsst.multi INFO: found job 17148243 in provider checkpoint 2024-03-17 04:09:34 proc_lsst.multi:201 [INFO] cancelling 17148243 on provider checkpoint proc_lsst.multi INFO: cancelling 17148243 on provider checkpoint 2024-03-17 04:09:34 proc_lsst.multi:146 [INFO] found job 17148244 in provider checkpoint proc_lsst.multi INFO: found job 17148244 in provider checkpoint 2024-03-17 04:09:34 proc_lsst.multi:201 [INFO] cancelling 17148244 on provider checkpoint proc_lsst.multi INFO: cancelling 17148244 on provider checkpoint 2024-03-17 04:09:34 proc_lsst.multi:146 [INFO] found job 17148245 in provider checkpoint proc_lsst.multi INFO: found job 17148245 in provider checkpoint 2024-03-17 04:09:34 proc_lsst.multi:201 [INFO] cancelling 17148245 on provider checkpoint proc_lsst.multi INFO: cancelling 17148245 on provider checkpoint 2024-03-17 04:09:35 proc_lsst.multi:146 [INFO] found job 17148246 in provider checkpoint proc_lsst.multi INFO: found job 17148246 in provider checkpoint 2024-03-17 04:09:35 proc_lsst.multi:201 [INFO] cancelling 17148246 on provider checkpoint proc_lsst.multi INFO: cancelling 17148246 on provider checkpoint 2024-03-17 04:09:35 proc_lsst.multi:146 [INFO] found job 17148247 in provider checkpoint proc_lsst.multi INFO: found job 17148247 in provider checkpoint 2024-03-17 04:09:35 proc_lsst.multi:201 [INFO] cancelling 17148247 on provider checkpoint proc_lsst.multi INFO: cancelling 17148247 on provider checkpoint 2024-03-17 04:09:35 proc_lsst.multi:146 [INFO] found job 17148248 in provider checkpoint proc_lsst.multi INFO: found job 17148248 in provider checkpoint 2024-03-17 04:09:35 proc_lsst.multi:201 [INFO] cancelling 17148248 on provider checkpoint proc_lsst.multi INFO: cancelling 17148248 on provider checkpoint 2024-03-17 04:09:35 proc_lsst.multi:146 [INFO] found job 17148249 in provider checkpoint proc_lsst.multi INFO: found job 17148249 in provider checkpoint 2024-03-17 04:09:35 proc_lsst.multi:201 [INFO] cancelling 17148249 on provider checkpoint proc_lsst.multi INFO: cancelling 17148249 on provider checkpoint 2024-03-17 04:09:36 proc_lsst.multi:146 [INFO] found job 17148250 in provider checkpoint proc_lsst.multi INFO: found job 17148250 in provider checkpoint 2024-03-17 04:09:36 proc_lsst.multi:201 [INFO] cancelling 17148250 on provider checkpoint proc_lsst.multi INFO: cancelling 17148250 on provider checkpoint 2024-03-17 04:09:36 proc_lsst.multi:146 [INFO] found job 17148251 in provider checkpoint proc_lsst.multi INFO: found job 17148251 in provider checkpoint 2024-03-17 04:09:36 proc_lsst.multi:201 [INFO] cancelling 17148251 on provider checkpoint proc_lsst.multi INFO: cancelling 17148251 on provider checkpoint 2024-03-17 04:09:36 proc_lsst.multi:146 [INFO] found job 17148252 in provider checkpoint proc_lsst.multi INFO: found job 17148252 in provider checkpoint 2024-03-17 04:09:36 proc_lsst.multi:201 [INFO] cancelling 17148252 on provider checkpoint proc_lsst.multi INFO: cancelling 17148252 on provider checkpoint 2024-03-17 04:09:37 proc_lsst.multi:146 [INFO] found job 17148253 in provider checkpoint proc_lsst.multi INFO: found job 17148253 in provider checkpoint 2024-03-17 04:09:37 proc_lsst.multi:201 [INFO] cancelling 17148253 on provider checkpoint proc_lsst.multi INFO: cancelling 17148253 on provider checkpoint 2024-03-17 04:09:37 proc_lsst.multi:146 [INFO] found job 17148254 in provider checkpoint proc_lsst.multi INFO: found job 17148254 in provider checkpoint 2024-03-17 04:09:37 proc_lsst.multi:201 [INFO] cancelling 17148254 on provider checkpoint proc_lsst.multi INFO: cancelling 17148254 on provider checkpoint 2024-03-17 04:09:37 proc_lsst.multi:146 [INFO] found job 17148255 in provider checkpoint proc_lsst.multi INFO: found job 17148255 in provider checkpoint 2024-03-17 04:09:37 proc_lsst.multi:201 [INFO] cancelling 17148255 on provider checkpoint proc_lsst.multi INFO: cancelling 17148255 on provider checkpoint 2024-03-17 04:09:38 proc_lsst.multi:146 [INFO] found job 17148256 in provider checkpoint proc_lsst.multi INFO: found job 17148256 in provider checkpoint 2024-03-17 04:09:38 proc_lsst.multi:201 [INFO] cancelling 17148256 on provider checkpoint proc_lsst.multi INFO: cancelling 17148256 on provider checkpoint 2024-03-17 04:09:38 proc_lsst.multi:146 [INFO] found job 17148257 in provider checkpoint proc_lsst.multi INFO: found job 17148257 in provider checkpoint 2024-03-17 04:09:38 proc_lsst.multi:201 [INFO] cancelling 17148257 on provider checkpoint proc_lsst.multi INFO: cancelling 17148257 on provider checkpoint 2024-03-17 04:09:38 proc_lsst.multi:146 [INFO] found job 17148258 in provider checkpoint proc_lsst.multi INFO: found job 17148258 in provider checkpoint 2024-03-17 04:09:38 proc_lsst.multi:201 [INFO] cancelling 17148258 on provider checkpoint proc_lsst.multi INFO: cancelling 17148258 on provider checkpoint 2024-03-17 04:09:39 proc_lsst.multi:146 [INFO] found job 17147931 in provider astro proc_lsst.multi INFO: found job 17147931 in provider astro 2024-03-17 04:09:39 proc_lsst.multi:201 [INFO] cancelling 17147931 on provider astro proc_lsst.multi INFO: cancelling 17147931 on provider astro 2024-03-17 04:09:39 proc_lsst.multi:146 [INFO] found job 17147932 in provider astro proc_lsst.multi INFO: found job 17147932 in provider astro 2024-03-17 04:09:39 proc_lsst.multi:201 [INFO] cancelling 17147932 on provider astro proc_lsst.multi INFO: cancelling 17147932 on provider astro 2024-03-17 04:09:39 proc_lsst.multi:146 [INFO] found job 17147933 in provider astro proc_lsst.multi INFO: found job 17147933 in provider astro 2024-03-17 04:09:39 proc_lsst.multi:201 [INFO] cancelling 17147933 on provider astro proc_lsst.multi INFO: cancelling 17147933 on provider astro 2024-03-17 04:09:40 proc_lsst.multi:146 [INFO] found job 17147934 in provider astro proc_lsst.multi INFO: found job 17147934 in provider astro 2024-03-17 04:09:40 proc_lsst.multi:201 [INFO] cancelling 17147934 on provider astro proc_lsst.multi INFO: cancelling 17147934 on provider astro 2024-03-17 04:09:40 proc_lsst.multi:146 [INFO] found job 17147935 in provider astro proc_lsst.multi INFO: found job 17147935 in provider astro 2024-03-17 04:09:40 proc_lsst.multi:201 [INFO] cancelling 17147935 on provider astro proc_lsst.multi INFO: cancelling 17147935 on provider astro 2024-03-17 04:09:40 proc_lsst.multi:146 [INFO] found job 17147936 in provider astro proc_lsst.multi INFO: found job 17147936 in provider astro 2024-03-17 04:09:40 proc_lsst.multi:201 [INFO] cancelling 17147936 on provider astro proc_lsst.multi INFO: cancelling 17147936 on provider astro 2024-03-17 04:09:40 proc_lsst.multi:146 [INFO] found job 17147937 in provider astro proc_lsst.multi INFO: found job 17147937 in provider astro 2024-03-17 04:09:40 proc_lsst.multi:201 [INFO] cancelling 17147937 on provider astro proc_lsst.multi INFO: cancelling 17147937 on provider astro 2024-03-17 04:09:41 proc_lsst.multi:146 [INFO] found job 17147938 in provider astro proc_lsst.multi INFO: found job 17147938 in provider astro 2024-03-17 04:09:41 proc_lsst.multi:201 [INFO] cancelling 17147938 on provider astro proc_lsst.multi INFO: cancelling 17147938 on provider astro 2024-03-17 04:09:41 proc_lsst.multi:146 [INFO] found job 17147939 in provider astro proc_lsst.multi INFO: found job 17147939 in provider astro 2024-03-17 04:09:41 proc_lsst.multi:201 [INFO] cancelling 17147939 on provider astro proc_lsst.multi INFO: cancelling 17147939 on provider astro 2024-03-17 04:09:41 proc_lsst.multi:146 [INFO] found job 17147940 in provider astro proc_lsst.multi INFO: found job 17147940 in provider astro 2024-03-17 04:09:41 proc_lsst.multi:201 [INFO] cancelling 17147940 on provider astro proc_lsst.multi INFO: cancelling 17147940 on provider astro 2024-03-17 04:09:42 proc_lsst.multi:146 [INFO] found job 17147941 in provider astro proc_lsst.multi INFO: found job 17147941 in provider astro 2024-03-17 04:09:42 proc_lsst.multi:201 [INFO] cancelling 17147941 on provider astro proc_lsst.multi INFO: cancelling 17147941 on provider astro 2024-03-17 04:09:42 proc_lsst.multi:146 [INFO] found job 17147942 in provider astro proc_lsst.multi INFO: found job 17147942 in provider astro 2024-03-17 04:09:42 proc_lsst.multi:201 [INFO] cancelling 17147942 on provider astro proc_lsst.multi INFO: cancelling 17147942 on provider astro 2024-03-17 04:09:42 proc_lsst.multi:146 [INFO] found job 17147943 in provider astro proc_lsst.multi INFO: found job 17147943 in provider astro 2024-03-17 04:09:42 proc_lsst.multi:201 [INFO] cancelling 17147943 on provider astro proc_lsst.multi INFO: cancelling 17147943 on provider astro 2024-03-17 04:09:43 proc_lsst.multi:146 [INFO] found job 17147944 in provider astro proc_lsst.multi INFO: found job 17147944 in provider astro 2024-03-17 04:09:43 proc_lsst.multi:201 [INFO] cancelling 17147944 on provider astro proc_lsst.multi INFO: cancelling 17147944 on provider astro 2024-03-17 04:09:43 proc_lsst.multi:146 [INFO] found job 17147945 in provider astro proc_lsst.multi INFO: found job 17147945 in provider astro 2024-03-17 04:09:43 proc_lsst.multi:201 [INFO] cancelling 17147945 on provider astro proc_lsst.multi INFO: cancelling 17147945 on provider astro 2024-03-17 04:09:43 proc_lsst.multi:146 [INFO] found job 17147946 in provider astro proc_lsst.multi INFO: found job 17147946 in provider astro 2024-03-17 04:09:43 proc_lsst.multi:201 [INFO] cancelling 17147946 on provider astro proc_lsst.multi INFO: cancelling 17147946 on provider astro 2024-03-17 04:09:44 proc_lsst.multi:146 [INFO] found job 17147947 in provider astro proc_lsst.multi INFO: found job 17147947 in provider astro 2024-03-17 04:09:44 proc_lsst.multi:201 [INFO] cancelling 17147947 on provider astro proc_lsst.multi INFO: cancelling 17147947 on provider astro 2024-03-17 04:09:44 proc_lsst.multi:146 [INFO] found job 17147948 in provider astro proc_lsst.multi INFO: found job 17147948 in provider astro 2024-03-17 04:09:44 proc_lsst.multi:201 [INFO] cancelling 17147948 on provider astro proc_lsst.multi INFO: cancelling 17147948 on provider astro 2024-03-17 04:09:44 proc_lsst.multi:146 [INFO] found job 17147950 in provider astro proc_lsst.multi INFO: found job 17147950 in provider astro 2024-03-17 04:09:44 proc_lsst.multi:201 [INFO] cancelling 17147950 on provider astro proc_lsst.multi INFO: cancelling 17147950 on provider astro 2024-03-17 04:09:44 proc_lsst.multi:146 [INFO] found job 17147951 in provider astro proc_lsst.multi INFO: found job 17147951 in provider astro 2024-03-17 04:09:44 proc_lsst.multi:201 [INFO] cancelling 17147951 on provider astro proc_lsst.multi INFO: cancelling 17147951 on provider astro 2024-03-17 04:09:45 proc_lsst.multi:146 [INFO] found job 17147952 in provider astro proc_lsst.multi INFO: found job 17147952 in provider astro 2024-03-17 04:09:45 proc_lsst.multi:201 [INFO] cancelling 17147952 on provider astro proc_lsst.multi INFO: cancelling 17147952 on provider astro 2024-03-17 04:09:45 proc_lsst.multi:146 [INFO] found job 17147953 in provider astro proc_lsst.multi INFO: found job 17147953 in provider astro 2024-03-17 04:09:45 proc_lsst.multi:201 [INFO] cancelling 17147953 on provider astro proc_lsst.multi INFO: cancelling 17147953 on provider astro 2024-03-17 04:09:45 proc_lsst.multi:146 [INFO] found job 17147954 in provider astro proc_lsst.multi INFO: found job 17147954 in provider astro 2024-03-17 04:09:45 proc_lsst.multi:201 [INFO] cancelling 17147954 on provider astro proc_lsst.multi INFO: cancelling 17147954 on provider astro 2024-03-17 04:09:46 proc_lsst.multi:146 [INFO] found job 17147955 in provider astro proc_lsst.multi INFO: found job 17147955 in provider astro 2024-03-17 04:09:46 proc_lsst.multi:201 [INFO] cancelling 17147955 on provider astro proc_lsst.multi INFO: cancelling 17147955 on provider astro 2024-03-17 04:09:46 proc_lsst.multi:146 [INFO] found job 17147956 in provider astro proc_lsst.multi INFO: found job 17147956 in provider astro 2024-03-17 04:09:46 proc_lsst.multi:201 [INFO] cancelling 17147956 on provider astro proc_lsst.multi INFO: cancelling 17147956 on provider astro 2024-03-17 04:09:46 proc_lsst.multi:146 [INFO] found job 17147957 in provider astro proc_lsst.multi INFO: found job 17147957 in provider astro 2024-03-17 04:09:46 proc_lsst.multi:201 [INFO] cancelling 17147957 on provider astro proc_lsst.multi INFO: cancelling 17147957 on provider astro 2024-03-17 04:09:47 proc_lsst.multi:146 [INFO] found job 17147958 in provider astro proc_lsst.multi INFO: found job 17147958 in provider astro 2024-03-17 04:09:47 proc_lsst.multi:201 [INFO] cancelling 17147958 on provider astro proc_lsst.multi INFO: cancelling 17147958 on provider astro 2024-03-17 04:09:47 proc_lsst.multi:146 [INFO] found job 17147959 in provider astro proc_lsst.multi INFO: found job 17147959 in provider astro 2024-03-17 04:09:47 proc_lsst.multi:201 [INFO] cancelling 17147959 on provider astro proc_lsst.multi INFO: cancelling 17147959 on provider astro 2024-03-17 04:09:47 proc_lsst.multi:146 [INFO] found job 17147960 in provider astro proc_lsst.multi INFO: found job 17147960 in provider astro 2024-03-17 04:09:47 proc_lsst.multi:201 [INFO] cancelling 17147960 on provider astro proc_lsst.multi INFO: cancelling 17147960 on provider astro 2024-03-17 04:09:48 proc_lsst.multi:146 [INFO] found job 17147961 in provider astro proc_lsst.multi INFO: found job 17147961 in provider astro 2024-03-17 04:09:48 proc_lsst.multi:201 [INFO] cancelling 17147961 on provider astro proc_lsst.multi INFO: cancelling 17147961 on provider astro 2024-03-17 04:09:48 proc_lsst.multi:146 [INFO] found job 17147977 in provider escience_a40 proc_lsst.multi INFO: found job 17147977 in provider escience_a40 2024-03-17 04:09:48 proc_lsst.multi:201 [INFO] cancelling 17147977 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147977 on provider escience_a40 2024-03-17 04:09:48 proc_lsst.multi:146 [INFO] found job 17147978 in provider escience_a40 proc_lsst.multi INFO: found job 17147978 in provider escience_a40 2024-03-17 04:09:48 proc_lsst.multi:201 [INFO] cancelling 17147978 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147978 on provider escience_a40 2024-03-17 04:09:48 proc_lsst.multi:146 [INFO] found job 17147979 in provider escience_a40 proc_lsst.multi INFO: found job 17147979 in provider escience_a40 2024-03-17 04:09:48 proc_lsst.multi:201 [INFO] cancelling 17147979 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147979 on provider escience_a40 2024-03-17 04:09:49 proc_lsst.multi:146 [INFO] found job 17147980 in provider escience_a40 proc_lsst.multi INFO: found job 17147980 in provider escience_a40 2024-03-17 04:09:49 proc_lsst.multi:201 [INFO] cancelling 17147980 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147980 on provider escience_a40 2024-03-17 04:09:49 proc_lsst.multi:146 [INFO] found job 17147981 in provider escience_a40 proc_lsst.multi INFO: found job 17147981 in provider escience_a40 2024-03-17 04:09:49 proc_lsst.multi:201 [INFO] cancelling 17147981 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147981 on provider escience_a40 2024-03-17 04:09:49 proc_lsst.multi:146 [INFO] found job 17147983 in provider escience_a40 proc_lsst.multi INFO: found job 17147983 in provider escience_a40 2024-03-17 04:09:49 proc_lsst.multi:201 [INFO] cancelling 17147983 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147983 on provider escience_a40 2024-03-17 04:09:50 proc_lsst.multi:146 [INFO] found job 17147984 in provider escience_a40 proc_lsst.multi INFO: found job 17147984 in provider escience_a40 2024-03-17 04:09:50 proc_lsst.multi:201 [INFO] cancelling 17147984 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147984 on provider escience_a40 2024-03-17 04:09:50 proc_lsst.multi:146 [INFO] found job 17147985 in provider escience_a40 proc_lsst.multi INFO: found job 17147985 in provider escience_a40 2024-03-17 04:09:50 proc_lsst.multi:201 [INFO] cancelling 17147985 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147985 on provider escience_a40 2024-03-17 04:09:50 proc_lsst.multi:146 [INFO] found job 17147986 in provider escience_a40 proc_lsst.multi INFO: found job 17147986 in provider escience_a40 2024-03-17 04:09:50 proc_lsst.multi:201 [INFO] cancelling 17147986 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147986 on provider escience_a40 2024-03-17 04:09:50 proc_lsst.multi:146 [INFO] found job 17147987 in provider escience_a40 proc_lsst.multi INFO: found job 17147987 in provider escience_a40 2024-03-17 04:09:50 proc_lsst.multi:201 [INFO] cancelling 17147987 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147987 on provider escience_a40 2024-03-17 04:09:51 proc_lsst.multi:146 [INFO] found job 17147988 in provider escience_a40 proc_lsst.multi INFO: found job 17147988 in provider escience_a40 2024-03-17 04:09:51 proc_lsst.multi:201 [INFO] cancelling 17147988 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147988 on provider escience_a40 2024-03-17 04:09:51 proc_lsst.multi:146 [INFO] found job 17147989 in provider escience_a40 proc_lsst.multi INFO: found job 17147989 in provider escience_a40 2024-03-17 04:09:51 proc_lsst.multi:201 [INFO] cancelling 17147989 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147989 on provider escience_a40 2024-03-17 04:09:51 proc_lsst.multi:146 [INFO] found job 17147992 in provider escience_a40 proc_lsst.multi INFO: found job 17147992 in provider escience_a40 2024-03-17 04:09:51 proc_lsst.multi:201 [INFO] cancelling 17147992 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147992 on provider escience_a40 2024-03-17 04:09:52 proc_lsst.multi:146 [INFO] found job 17147993 in provider escience_a40 proc_lsst.multi INFO: found job 17147993 in provider escience_a40 2024-03-17 04:09:52 proc_lsst.multi:201 [INFO] cancelling 17147993 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147993 on provider escience_a40 2024-03-17 04:09:52 proc_lsst.multi:146 [INFO] found job 17147994 in provider escience_a40 proc_lsst.multi INFO: found job 17147994 in provider escience_a40 2024-03-17 04:09:52 proc_lsst.multi:201 [INFO] cancelling 17147994 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147994 on provider escience_a40 2024-03-17 04:09:52 proc_lsst.multi:146 [INFO] found job 17147995 in provider escience_a40 proc_lsst.multi INFO: found job 17147995 in provider escience_a40 2024-03-17 04:09:52 proc_lsst.multi:201 [INFO] cancelling 17147995 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147995 on provider escience_a40 2024-03-17 04:09:52 proc_lsst.multi:146 [INFO] found job 17147996 in provider escience_a40 proc_lsst.multi INFO: found job 17147996 in provider escience_a40 2024-03-17 04:09:52 proc_lsst.multi:201 [INFO] cancelling 17147996 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147996 on provider escience_a40 2024-03-17 04:09:53 proc_lsst.multi:146 [INFO] found job 17147997 in provider escience_a40 proc_lsst.multi INFO: found job 17147997 in provider escience_a40 2024-03-17 04:09:53 proc_lsst.multi:201 [INFO] cancelling 17147997 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147997 on provider escience_a40 2024-03-17 04:09:53 proc_lsst.multi:146 [INFO] found job 17147998 in provider escience_a40 proc_lsst.multi INFO: found job 17147998 in provider escience_a40 2024-03-17 04:09:53 proc_lsst.multi:201 [INFO] cancelling 17147998 on provider escience_a40 proc_lsst.multi INFO: cancelling 17147998 on provider escience_a40 2024-03-17 04:09:53 proc_lsst.multi:146 [INFO] found job 17148000 in provider escience_a40 proc_lsst.multi INFO: found job 17148000 in provider escience_a40 2024-03-17 04:09:53 proc_lsst.multi:201 [INFO] cancelling 17148000 on provider escience_a40 proc_lsst.multi INFO: cancelling 17148000 on provider escience_a40 2024-03-17 04:09:54 proc_lsst.multi:146 [INFO] found job 17147962 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147962 in provider escience_rtx6k 2024-03-17 04:09:54 proc_lsst.multi:201 [INFO] cancelling 17147962 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17147962 on provider escience_rtx6k 2024-03-17 04:09:54 proc_lsst.multi:146 [INFO] found job 17147963 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147963 in provider escience_rtx6k 2024-03-17 04:09:54 proc_lsst.multi:201 [INFO] cancelling 17147963 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17147963 on provider escience_rtx6k 2024-03-17 04:09:54 proc_lsst.multi:146 [INFO] found job 17147964 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147964 in provider escience_rtx6k 2024-03-17 04:09:54 proc_lsst.multi:201 [INFO] cancelling 17147964 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17147964 on provider escience_rtx6k 2024-03-17 04:09:55 proc_lsst.multi:146 [INFO] found job 17147965 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147965 in provider escience_rtx6k 2024-03-17 04:09:55 proc_lsst.multi:201 [INFO] cancelling 17147965 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17147965 on provider escience_rtx6k 2024-03-17 04:09:55 proc_lsst.multi:146 [INFO] found job 17147966 in provider escience_rtx6k proc_lsst.multi INFO: found job 17147966 in provider escience_rtx6k 2024-03-17 04:09:55 proc_lsst.multi:201 [INFO] cancelling 17147966 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17147966 on provider escience_rtx6k 2024-03-17 04:09:55 proc_lsst.multi:146 [INFO] found job 3544 in provider local proc_lsst.multi INFO: found job 3544 in provider local 2024-03-17 04:09:55 proc_lsst.multi:201 [INFO] cancelling 3544 on provider local proc_lsst.multi INFO: cancelling 3544 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 3544 parsl.providers.local.local WARNING: Failed to kill PID: 3544 and child processes on local 2024-03-17 04:09:56 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 513.8836 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210909_B0c_science#step0_20240317T040120Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 513.8914 seconds; current memory usage: 0.323 Gibyte, delta: 0.019 Gibyte, peak delta: 0.020 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 531.2944 seconds; current memory usage: 0.323 Gibyte, delta: 0.186 Gibyte, peak delta: 0.188 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.324 Gibyte (main), 0.324 Gibyte (largest child process) Run Id: None Run Name: DEEP_20210909_B0c_science#step0_20240317T040120Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 8m53.873s user 1m41.683s sys 0m58.340s